Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

"A scrappy little startup working out of a DC garage that completely influenced the course of how the web-facing portions of the Affordable Care Act were to be implemented"

For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-heavy situation, they are somewhat limited in how much caching will help.

One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. That whole part didn't need to be database driven at all. The data could have been stored in redis and they could have used javascript to filter it based on a form. That would have been ridiculously fast. They also could have prerendered all the plan list possibilities and stored those in varnish. That also would have been ridiculously fast. My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.

Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data.

Sharding your database is something that is probably possible, and in the case of healthcare.gov, they probably could have had totally separate infrastructure on a per-state basis that would have made scaling a lot easier than putting everything on the same database. Also, put reporting and things that aren't mission critical on a slave database. The last thing you want is a reporting job bringing down the live site in the background.

Getting good hardware with fast IO is going to save a lot of developer time required to scale things. Using fast SSD's is probably the easiest win to speed up your database. Developer time costs a lot more than hardware and giving yourself cheap headroom up front gives you breathing room on launch.

Performance testing is also something worth doing, but the tricky part is until you roll out, it is hard to know exactly where the hotspots are going to be. In this case, new user signup would be the obvious place to test, so they probably should have tested up to the limits of their servers and tried to extrapolate an expected number of users and maybe increased that by 1.5x or something to have some leeway.

On a rollout where you don't know what you are getting into user wise, being on a cloud where you can scale out fast as demand requires is something worth doing. They could have saved a lot of bad press and headache by being on the cloud initially and migrating to less hardware after the initial peak died down.

There are a ton of little things like that you have to think about if you are dealing with massive scale. I don't know if the engineers building healthcare.gov had ever dealt with something like this before, but I'm sure they're learning these lessons now.

In Maryland, our exchange website is poorly designed and written, just on the frontend (I obviously can't see the backend), but at the same time, all of the frontend issues could be fixed with a clever caching scheme.

Where the real bottleneck almost certainly lies is when the system takes your user submitted data and has to post it into what is surely an old, legacy federal system so that it can verify your identity. That old legacy system may have seen upgrades in preparation for this, but probably not -- even so, there's simply no way to prepare for the onslaught of users accessing (indirectly) what was certain to have been an isolated, government-only database in the past.

Gov't contracting is, in economic terms, a rent-seeking business. The best and brightest in a given contracting firm are dedicated to pursuing new work. I work for a contractor, and my fun IT projects are all proof of concept items. Stellar execution simply isn't rewarded. The gov't puts idiots in charge of things: not idiots in the sense that they aren't smart, but idiots in the sense that they have no expertise in anything. They are hired for paper qualifications (ever heard of a Project Management Professional cert? It's what they hire, and its useless), and the bright/motivated ones move on to interesting work, perhaps in other parts of the gov't, or the private sector. I built a great tool, using all open source technology for the gov't once. I built it in 3 weeks (it is now officially gov't owned software), and it replaced an $80K piece of software (commercial off the shelf) that was inadequate for the task. My reward was complaints that the security people weren't familiar with Postgres, and an immediate request to migrate to Oracle. There was no functional reason for this, but I complied after the gov't shelled out huge sums of money for Oracle license. Why? Because paper pushing idiots in the gov't are in charge of IT security, and their credentials for the job are based entirely on passing certification tests. Their entire incentivization at their job is to minimize their own workload, rather than maximize the happiness of their gov't customers.

Do you think anybody who could even guage whether proper testing was being performed or not was in charge of this project from the gov't level? Of course not. If they had, it would have been properly tested.

I'm mad as hell about this because I want the ACA to succeed. I don't think its ideal, but its vastly better than the existing system of hospitals as clinics for the uninsured hordes.

The Federal gov't has an awful hiring process, and until it is fixed, you are lucky to encounter competent and driven individuals within it. You are foolish to expect it.

1. Slow Query Log 2. Grep commands