Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The US Census now has an API

census.gov

Post reply on HN
Print view View original

11–20 of 42 posts

Re: The US Census now has an API

#11

Post by [deleted] »

tantalor wrote: post #7
jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

I also found this odd, but I can think of two reasons, 1. CSV is not a single well-defined format. Some people use rfc4180, but not everybody does. 2. JSON is easier to parse than CSV for JavaScript clients, i.e., browsers. On the other hand, JSON is not a subset of JavaScript.

[deleted]

Re: The US Census now has an API

#12

Post by [deleted] »

pnathan wrote: post #8
jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

It's very simple to transform that kind of data into $your_preferred_structure. It's also a very agnostic format.

[deleted]

Re: The US Census now has an API

#13

Post by bcherry »

jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

This doesn't seem unreasonable.

1) Whether it's user-friendly or not, it's still JSON, which makes JSON-P possible (they support JSON-P).

2) It probably mirrors the way they store their data (in tables, whether SQL or Access or Excel, doesn't really matter).

3) It's more compact than traditional JSON, which means less bandwidth, which means less cost. Keep in mind that this is essentially a not-for-profit API from a not-for-profit organization with the worst possible budgeting scenario. Yes, Gzip would basically eliminate this benefit, but they don't have gzip enabled and enabling it may be difficult or impossible under whatever constraints they operate under.

Also, it's entirely possible that this API has existed privately for a long, long time in a CSV format, and they simply made a minor enhancement to make it JSON and JSON-P compatible and open up the endpoints to the public.

Re: The US Census now has an API

#14

Post by jack-r-abbit »

bcherry wrote: post #13
jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

This doesn't seem unreasonable. 1) Whether it's user-friendly or not, it's still JSON, which makes JSON-P possible (they support JSON-P). 2) It probably mirrors the way they store their data (in tables, whether SQL or Access or Excel, doesn't really matter). 3) It's more compact than traditional JSON, which means less bandwidth, which means less cost. Keep in mind that this is essentially a not-for-profit API from a…

#3 seems a likely reason. and like I said, having a harder to work with data structure is still better than nothing. and it is free.

Re: The US Census now has an API

#15

Post by adelevie »

If anyone is interested, the FCC has a fairly comprehensive list of Developers pages for other federal agencies: http://www.fcc.gov/developers (on the right column).

Also, for further reading on the topic of .gov APIs, http://ben.balter.com/2012/06/02/publishing-government-data-... is a great start.

Re: The US Census now has an API

#16

Post by peter_l_downs »

bcherry wrote: post #13
jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

This doesn't seem unreasonable. 1) Whether it's user-friendly or not, it's still JSON, which makes JSON-P possible (they support JSON-P). 2) It probably mirrors the way they store their data (in tables, whether SQL or Access or Excel, doesn't really matter). 3) It's more compact than traditional JSON, which means less bandwidth, which means less cost. Keep in mind that this is essentially a not-for-profit API from a…

All good explanations, but I can't help but wish they had chosen a format like:

  {
    P0010001 : [
      '710231',
      '4779736',
      // ... etc.
    ],
    NAME : [
      'Alaska',
      'Alabama',
      // ... etc.
    ],
    state: [
      '02',
      '01,
      // ... etc.
    ],
    // ... etc.
  }

Re: The US Census now has an API

#17

Post by jlarocco »

systematical wrote: post #4
DrJokepu wrote: post #2

Wouldn't this be better served by a downloadable SQLite file rather than a web service?

It's database agnostic this way and every language supports JSON.

Partly to play devil's advocate and partly out of curiosity, do you know of a language with a JSON library but no SQLite library?

Re: The US Census now has an API

#18

Post by smacktoward »

DrJokepu wrote: post #2

Wouldn't this be better served by a downloadable SQLite file rather than a web service?

The Census Bureau has made their data available for download for a looong time, both over the Web (including a bunch of different interfaces for slicing out just the data you want -- see http://www.census.gov/main/www/access.html) and FTP (http://www2.census.gov/) All the stuff that's available via the Web service is already available via download, along with a lot more. They don't provide it in SQLite format AFAIK, but they do offer a range of other formats that should be easy to get into SQLite if you wish.

If there's any problem with their downloadable data offerings, it's that they have so many of them that it can be difficult to figure out exactly which one has the data you're looking for.

Re: The US Census now has an API

#19

Post by Yen »

DrJokepu wrote: post #2

Wouldn't this be better served by a downloadable SQLite file rather than a web service?

for you, and your repliers:

this data is already available as a downloadable file. Check out ftp://ftp.census.gov/ It's CSV, and it's kinda ugly-looking, but it's available.

The 'full' data set is ...rather huge. The full data set for any particular report is rather huge.

The advantage of this API, I expect, is it allows people to make quick, machine readable queries for useful subsets of the data. If you want a JavaScript mashup that tells you the number of people who live on your block, use this. If you want to know the population of every block everywhere for data-crunching purposes, just download the CSV.

Re: The US Census now has an API

#20

Post by lancefisher »

jack-r-abbit wrote: post #5

> The response for all queries is formatted as a two dimensional JSON array where the first row provides column names and subsequent rows provide data values. Hmmm... looking at the example of this I can't help but think there has got to be a better way. This is more just a standard CSV (first row is header, all other rows are data). Using JSON for data formatted as such is kind of a waste of JSON. If you pass that e…

Factual.com serves their data like this too. I thought it was odd at first, but it does save a lot of bytes on the wire. I just parsed it with JSON.parse() and used underscore to map to my object format. You don't really even need underscore for that. It's just a matter of mapping an array of rows to an array of objects. It's nice there is an API now.
Post reply on HN