Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The US Census now has an API

census.gov

Post reply on HN
Print view View original

31–40 of 42 posts

Re: The US Census now has an API

#31

Post by tzs »

peter_l_downs wrote: post #16
bcherry wrote: post #13

Earlier quoted context omitted.

This doesn't seem unreasonable. 1) Whether it's user-friendly or not, it's still JSON, which makes JSON-P possible (they support JSON-P). 2) It probably mirrors the way they store their data (in tables, whether SQL or Access or Excel, doesn't really matter). 3) It's more compact than traditional JSON, which means less bandwidth, which means less cost. Keep in mind that this is essentially a not-for-profit API from a…

All good explanations, but I can't help but wish they had chosen a format like: { P0010001 : [ '710231', '4779736', // ... etc. ], NAME : [ 'Alaska', 'Alabama', // ... etc. ], state: [ '02', '01, // ... etc. ], // ... etc. }

I'm having trouble thinking of any situation where that would be more convenient or useful than the format they are using now. Where would your proposed format be an advantage?

Re: The US Census now has an API

#32

Post by tzs »

jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

With their format, it is an easy matter to mechanically turn the first item into:

   INSERT INTO table ('P0010001','NAME','state') VALUES (?,?,?)
and then loop through the remaining items using that to insert them into a database.

Re: The US Census now has an API

#33

Post by blt »

It's pretty cool that they are putting up the machine time to run queries on the data. I wonder how long it will last if someone makes a popular web app using the census API.

Re: The US Census now has an API

#34

Post by mayel »

10dpd wrote: post #30

Has anyone considered possible use-cases for this dataset? It's great to get excited about the data format, but actual use-cases currently seem a little limited to me. Of course this will hopefully change as more and more datasets become available.

Here is a quick example I put together this weekend (using data from their Excel files, not the API - didn't know about it!) http://babelverse.com/linguistics/funfacts/

Looking forward to many more visualizations and analysis thanks to the API :)

Re: The US Census now has an API

#36

Post by jackcviers3 »

jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

One of the talks given at fluent conf (on Google's feedback canvas screenshot system) last week talked about how moving data from object-based storage to array-based storage saved serialization time and memory use. I would guess that is why they did it this way.

Re: The US Census now has an API

#37

Post by yahelc »

alister wrote: post #25

You need to request a "key" (essentially a user ID) before you can make queries. I can imagine only 3 reasons for this: (1) they want to make you agree to the Terms & Conditions, (2) it gives a way to choke off a DDOS attack that makes repeated complex queries, or (3) the census bureau wants to track how you are using its service. The key is indeed easy to get, but I observe that Google would have exactly the same co…

Google and the Census both don't require individual users to signup to access their data.

However, both require that their users signup for an API key to do automated querying. (In fact, Google charges for its Web Search API now.) It's pretty standard for any API to want to be able to identify and potentially meter usage.

Re: The US Census now has an API

#38

Post by josephcooney »

DrJokepu wrote: post #2

Wouldn't this be better served by a downloadable SQLite file rather than a web service?

As other have already commented, this is already available as a download. You can even fire up an EC2 instance with the data pre-downloaded http://aws.amazon.com/datasets/Economics?_encoding=UTF8&...

Re: The US Census now has an API

#39

Post by fennecfoxen »

chernevik wrote: post #27
alister wrote: post #25

You need to request a "key" (essentially a user ID) before you can make queries. I can imagine only 3 reasons for this: (1) they want to make you agree to the Terms & Conditions, (2) it gives a way to choke off a DDOS attack that makes repeated complex queries, or (3) the census bureau wants to track how you are using its service. The key is indeed easy to get, but I observe that Google would have exactly the same co…

(3B) Census data on some topics can move markets. If someone hacks to get a five minute lead time, the key policy gives them a start on figuring out who and maybe how.

Right. Because if I hack a website I'm totally going to log in with a totally real set of credentials traceable back to me.

Besides, this is the 2010 American Community Survey. Pretty sure it's moved all the markets it's going to move by now.

Re: The US Census now has an API

#40

Post by fennecfoxen »

awaythrow wrote: post #29
Yen wrote: post #19

Earlier quoted context omitted.

for you, and your repliers: this data is already available as a downloadable file. Check out ftp://ftp.census.gov/ It's CSV, and it's kinda ugly-looking, but it's available. The 'full' data set is ...rather huge. The full data set for any particular report is rather huge. The advantage of this API, I expect, is it allows people to make quick, machine readable queries for useful subsets of the data. If you want a Java…

Excellent. Much easier than requesting API keys and parsing JSON. But that's just my opinion. CSV is beautiful. Simple and easy to manipulate into some other format if desired.

CSV is beautiful - for numbers. When you invoke strings, and quoting, and whatnot, it goes downhill fast. :)
Post reply on HN