Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

blog.smartbear.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#41

Post by dragonwriter »

jrochkind1 wrote: post #38
programminggeek wrote: post #13

I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-he…

> if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. One question would be whether they hired people/firms who had had to deal with that level of scale before. I'd assume not. But apparently they hired someone who didn't even know enough to know that they were not qualified to do it.

One side effect of the complexity of government contracting processes (designed, in principle, to ensure that the government doesn't get taken advantage of by either internal or external actors) is that the people who get awarded government contracts are often the people with the most skill in negotiating the government contracting process, not the people with the most skill in the problem domain.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#42

Post by shamshiel »

DennisP wrote: post #39
programminggeek wrote: post #13

I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-he…

What's surprising to me at this point is that nobody has published the actual plans, separately from this website. Best thing the government could do right now is get those plans and their rates out to news organizations, so people can see them without going to healthcare.gov, which could go back to being just the enrollment application that they seem to have intended.

The website includes the plans with the subsidies, which is why they require you to sign up and answer a bunch of questions. I believe there's a lot of verification of subsidy eligibility going on in the background. Just having the rates out there would make it appear that the plans are much more expensive than they will be for most people.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#43

Post by shamshiel »

DennisP wrote: post #33
coldcode wrote: post #23

Earlier quoted context omitted.

STATIC site content. Not the part being hammered by people signing up. People keep confusing the two.

The plans should be static site content. You shouldn't have to initiate the sign-up process just to see what's available.

Seeing what's available without reliable information on your actual costs (including subsidies) doesn't accomplish much.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#44

Post by shamshiel »

mgkimsal wrote: post #36
jasonpeacock wrote: post #29

Regardless of any technological or architectural decisions, simple load testing before launch would have prevented this. 1. Load test, discover issues. 2. Fix issues. 3. Repeat steps 1-2 until no more issues. That is the flabbergasting part. Nobody actually tried applying real load to the system before release? It fell over so easily and quickly the only assumption is either no loadtesting was performed, the results…

The official story right now is that they underestimated. Medicaid has had at max 30k users; they doubled the estimate for hc.gov, but instead of ~50k users they're seeing 250k. From what I'd read (in a story linked from this hn page) they did test for load, but nowhere near 200k users. FWIW, I don't think the issues right now are load so much. I think there's some insanely bad logical pieces that are messing things…

I immediately get an error page after logging and and trying to view plans. That could be load but it certainly feels like some kind of logical error to me.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#46

Post by mgkimsal »

shamshiel wrote: post #44
mgkimsal wrote: post #36

Earlier quoted context omitted.

The official story right now is that they underestimated. Medicaid has had at max 30k users; they doubled the estimate for hc.gov, but instead of ~50k users they're seeing 250k. From what I'd read (in a story linked from this hn page) they did test for load, but nowhere near 200k users. FWIW, I don't think the issues right now are load so much. I think there's some insanely bad logical pieces that are messing things…

I immediately get an error page after logging and and trying to view plans. That could be load but it certainly feels like some kind of logical error to me.

Exactly. They've got the 'waiting room' technique to slow down apps/logins. The login process, once done, is quite fast in getting me to a blank page. (https://www.healthcare.gov/marketplace/auth/userprofile page in question)

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#47

Post by jrochkind1 »

Pxtl wrote: post #27
camus wrote: post #25

Earlier quoted context omitted.

Problem is your ORM might generate 300 fast queries to get 300 records where 1 would be faster than 300. using an ORM make things tricky since you basically surrender most of Db control . Scaling databases is hard,developpers like ORMs , but ORMs make scaling databases even harder.

The big thing with an ORM is that you must keep those lazy loads in mind when you work with it. Lazy loads are the big place where you get "300 queries instead of 1". ORMs aren't bad on principal, imho... but they give you more than enough rope to hang yourself while promising a magical efficient user-friendly rope.

I wonder why most ORM's don't let you turn off lazy loads -- "only fetch when I ask for a fetch, if I try to access something without having asked you to fetch it first -- raise an exception!" Just as an option.

This would not be a particularly difficult feature for an ORM to implement.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#48

Post by DennisP »

shamshiel wrote: post #42
DennisP wrote: post #39

Earlier quoted context omitted.

What's surprising to me at this point is that nobody has published the actual plans, separately from this website. Best thing the government could do right now is get those plans and their rates out to news organizations, so people can see them without going to healthcare.gov, which could go back to being just the enrollment application that they seem to have intended.

The website includes the plans with the subsidies, which is why they require you to sign up and answer a bunch of questions. I believe there's a lot of verification of subsidy eligibility going on in the background. Just having the rates out there would make it appear that the plans are much more expensive than they will be for most people.

Several news organizations have already made subsidy calculators. They just don't have specific plan details.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#49

Post by DennisP »

shamshiel wrote: post #43
DennisP wrote: post #33

Earlier quoted context omitted.

The plans should be static site content. You shouldn't have to initiate the sign-up process just to see what's available.

Seeing what's available without reliable information on your actual costs (including subsidies) doesn't accomplish much.

Premiums vary based on region, age, and smoking status. It's not so much information that you couldn't handle it with static pages.

Subsidy is based on income, and it's simple enough that NPR and others have already build subsidy calculators. Put that in javascript and you've still got static pages, running calculations client-side.

Re: Could Better Testing Practices Have Prevented the Healthcare.gov Defects?

#50

Post by snowwrestler »

dragonwriter wrote: post #41
jrochkind1 wrote: post #38

Earlier quoted context omitted.

> if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have. One question would be whether they hired people/firms who had had to deal with that level of scale before. I'd assume not. But apparently they hired someone who didn't even know enough to know that they were not qualified to do it.

One side effect of the complexity of government contracting processes (designed, in principle, to ensure that the government doesn't get taken advantage of by either internal or external actors) is that the people who get awarded government contracts are often the people with the most skill in negotiating the government contracting process, not the people with the most skill in the problem domain.

We know who built Healthcare.gov; it was Development Seed. There were a bunch of stories about it earlier this year. Development Seed also built and runs MapBox, so they should have some idea how to run a high volume web service.
Post reply on HN