IBM's reputation at risk in wake of census bungle

I'd wager there were other complexities purely from the govt side. When healthcare.gov launched and crashed, it was uncovered that the final definition of the laws and rules were still coming weeks before the launch and had changed dozens of times throughout.

From a purely technical perspective, there's no way to know if something "works" if it hasn't been defined or - worse! - is constantly being redefined on the fly.

A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different contractors. We needed a way to "securely" show a picture, aka not using the pre-installed image showing app. So IBM spent tens of thousands of dollars to build a new Android app that displayed a picture passed to it via an intent. I told our product guy that I could build it in an hour for free, but he said it wasn't our problem. In our twice-weekly status calls, we would ask about when the "image showing app" would arrive, and we heard all about the minutiae of building that app, from getting a PM, to finding a partner in India, to QA, to issues with the app, back to the contractor, and finally to us, where it was buggy as hell and never worked right. Completely mind boggling, and very enlightening about how IBM (and I'm sure other big vendors) operate.

> that barely seems like enough to pay for the hardware to deal with 10+ million visitors in one evening.

Really?

The specs: The online census form was a pretty simple client side app. I'm going to say, I could whack something together in much less than 200k of JS.

For each household the site needs to:

1. Serve the client side app. The app had no images beside the ABS header. I'll be generous and say 200k of static assets including the client side JS.

2. Accept & validate the code households received on their postal forms

3. Save the final result (a single record) back to the databases. A single append operation. Being generous, 20k of JSON data per household.

When I filled in the census on Thursday night I noticed it made no effort to normalize addresses or anything else. Umless I'm missing something (or IBM turned off half the features after the DOS) the site was that simple.

1. 200k of static assets * 10 million households = 2TB of traffic. AWS would serve 2TB of traffic out of cloudfront for a whopping $200. If you don't trust 'the public cloud' your self-run data center could serve a tiny payload which downloaded the JS from cloudfront and verified its hash before running it. Or just serve all 2TB of traffic directly.

2. Accepting the unique code on the posted form is easy - just have a few servers which store the database of all posted codes in RAM and respond with 'yes' or 'no'. The codes for 10M households will fit in less than a gig of ram.

2. Saving the final result is pretty trivial. You need to accept 200 gigs of JSON data in an append-only journal. The peak write load would be around 4 million requests / hour, or cumulative ~1000 writes per second, or ~20MB/second of data. For an append-only store backed by an SSD this isn't much load at all. If the numbers weren't so low I'd suggest using a hand-written journal, but honestly most off the shelf databases could handle this write load without breaking a sweat. And you could easily shard it if you wanted to, and just aggregate back the data after its been collected. (You might want to actively geo-replicate the data immediately too, while its being collected just in case something happens).

In total, I'd feel comfortable running this entire site out of a few dedicated machines in just about any data center in the country. If you're feeling paranoid geo-replicate it, and actively stream received census submissions to a couple offsite backup locations. Total whiteboarding time: about 20 minutes. Might make a good interview question.

Of course IBM was also paid for consultancy time, and engineering time to design & implement the site itself, but even factoring in healthy margins I can't help but feel our government was ripped off by a factor of ~10-20x.