Could Better Testing Practices Have Prevented the Healthcare.gov Defects? • Hacker News Board

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#31

Post by RougeFemme » Tue, Oct 08, 2013, 6:32 PM UTC

protomyth wrote: post #20

Setting aside the politics, I really don't think it could have been prevented. The specs were not exactly realistic on number of users at a time. Heck, even Apple still gets slammed and they know its coming. Plus, the experience just isn't in the DC contracting community. They can do websites, but not high availability transactional. If we were talking back-ends, then yes they have high transaction experience, but no…

But isn't the DC contracting community involved in other high availability transactional systems? For example, isn't one of the ACH systems a government or pseduo-government system?

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#32

Post by protomyth » Tue, Oct 08, 2013, 6:34 PM UTC

RougeFemme wrote: post #31

protomyth wrote: post #20

Setting aside the politics, I really don't think it could have been prevented. The specs were not exactly realistic on number of users at a time. Heck, even Apple still gets slammed and they know its coming. Plus, the experience just isn't in the DC contracting community. They can do websites, but not high availability transactional. If we were talking back-ends, then yes they have high transaction experience, but no…

But isn't the DC contracting community involved in other high availability transactional systems? For example, isn't one of the ACH systems a government or pseduo-government system?

Oh yes they are, but not anything with a web-front end. They do really well with mainframe and back-end, just don't expect anything that has to touch a web stack.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#33

Post by DennisP » Tue, Oct 08, 2013, 6:40 PM UTC

coldcode wrote: post #23

chintan wrote: post #11

http://blog.netizencorp.com/2013/10/06/how-a-scrappy-startup... "A scrappy little startup working out of a DC garage that completely influenced the course of how the web-facing portions of the Affordable Care Act were to be implemented"

STATIC site content. Not the part being hammered by people signing up. People keep confusing the two.

The plans should be static site content. You shouldn't have to initiate the sign-up process just to see what's available.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#34

Post by DennisP » Tue, Oct 08, 2013, 6:43 PM UTC

jasonpeacock wrote: post #29

Regardless of any technological or architectural decisions, simple load testing before launch would have prevented this. 1. Load test, discover issues. 2. Fix issues. 3. Repeat steps 1-2 until no more issues. That is the flabbergasting part. Nobody actually tried applying real load to the system before release? It fell over so easily and quickly the only assumption is either no loadtesting was performed, the results…

They probably load-tested based on the number of sign-ups they expected, and that estimate may even have been accurate. They don't seem to have realized that lots more people would be interested in what's available, even if they aren't ready to enroll. If they had realized that, you wouldn't have to "apply" before seeing the plans, and the plans could have been static content, distributed all over a CDN by now.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#35

Post by jrochkind1 » Tue, Oct 08, 2013, 6:44 PM UTC

JPKab wrote: post #17

jrochkind1 wrote: post #9

Sure, they could have been prevented. By spending more money. Isn't that always the answer? Creating a website that can handle, well, nearly the entire country, out of the gate, is not trivial. (Facebook etc got to scale up to that level, they didn't need to do it out of the gate). But it also can be done. How? Well, you hire real experts in doing this sort of thing. And you give them enough calendar time, and enough…

Your opinion shouldn't be high. There are talented IT people working at government contractors, but they aren't going to get put where they are needed, because the incentives for contractors are not performing the work at a top-notch level. Gov't contracting is, in economic terms, a rent-seeking business. The best and brightest in a given contracting firm are dedicated to pursuing new work. I work for a contractor, a…

I am not optimistic that they'll get it working by the deadline, sadly. Are you?

I mean, let's start admitting that it's _not_ easy to make a site that can handle that level of traffic. It's do-able, but it's not easy.

Once they have it handling the proper traffic, only _then_ are they going to start finding out about all the bugs that are actually logic related instead of load/traffic related. Which I'm sure there are plenty of, because all software has bugs, and because the QA for those (including integration with IRS system or whatever?) isn't trivial either, and I'm sure they didn't do any better with it then they did with preparing for the load.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#36

Post by mgkimsal » Tue, Oct 08, 2013, 6:45 PM UTC

jasonpeacock wrote: post #29

Regardless of any technological or architectural decisions, simple load testing before launch would have prevented this. 1. Load test, discover issues. 2. Fix issues. 3. Repeat steps 1-2 until no more issues. That is the flabbergasting part. Nobody actually tried applying real load to the system before release? It fell over so easily and quickly the only assumption is either no loadtesting was performed, the results…

The official story right now is that they underestimated. Medicaid has had at max 30k users; they doubled the estimate for hc.gov, but instead of ~50k users they're seeing 250k. From what I'd read (in a story linked from this hn page) they did test for load, but nowhere near 200k users.

FWIW, I don't think the issues right now are load so much. I think there's some insanely bad logical pieces that are messing things up, and possibly have corrupted accounts that were created early on. I base this on a few things.

1. The "username" requirements.

The username is case sensitive. Choose a username that is 6-74 characters long and must contain a lowercase or capital letter, a number, or one of these symbols _.@/-

Really? I know some systems use case-sensitive usernames, but given that you're already forcing some odd characters and whatnot in there, why not normalize to lowercase? This just feels like it's going to cause more support problems (mobile safari automatically uppercasing a username that should be lowercased, etc).

Also... the English description of that username is nowhere near intuitive. "A lowercase or capital letter". As opposed to what other kind of letter?

2. Multiple accounts with a single email address.

I've been able to 'successfully' register multiple different username accounts with the same email address. When I do a password reset based on email (the couple times it worked) which username was I resetting the password for? I say 'successfully' because no login attempt has ever worked. And now trying to register yet another username with the same email address doesn't work, but the error message is so vague ("there was a problem" IIRC) that I can't tell if that was a factor or not.

OK... 2 things. That's about as far as I've been able to see in to the system so far, so that's all I can judge it on. But it seems that they've probably allowed some logical inconsistencies in my own signups that may be causing more problems now, and I don't think they are related to load.

Of course, I could be 100% wrong, but it doesn't feel like load is the culprit right now.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#37

Post by [deleted] » Tue, Oct 08, 2013, 6:45 PM UTC

[deleted]

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#38

Post by jrochkind1 » Tue, Oct 08, 2013, 6:45 PM UTC

programminggeek wrote: post #13

I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-he…

> if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have.

One question would be whether they hired people/firms who had had to deal with that level of scale before.

I'd assume not.

But apparently they hired someone who didn't even know enough to know that they were not qualified to do it.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#39

Post by DennisP » Tue, Oct 08, 2013, 6:46 PM UTC

programminggeek wrote: post #13

I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-he…

What's surprising to me at this point is that nobody has published the actual plans, separately from this website. Best thing the government could do right now is get those plans and their rates out to news organizations, so people can see them without going to healthcare.gov, which could go back to being just the enrollment application that they seem to have intended.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#40

Post by jrochkind1 » Tue, Oct 08, 2013, 6:47 PM UTC

aganders3 wrote: post #18

programminggeek wrote: post #13

I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-he…

One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. ... My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth. That's all I wanted to do (I have employer-based coverage for now). I turned back once I realized you nee…

I wonder if the list of available plans is customized, not everyone has access to every plan, and it can't show you the list of plans available to you until it knows who you are.