Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

content.time.com

Post reply on HN
Print view View original

111–120 of 259 posts

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#111

Post by saraid216 »

dragontamer wrote: post #31
jobu wrote: post #18

Wow! They weren't even caching database calls: HealthCare.gov had been constructed so that every time a user had to get information from the website's vast database, the website had to make what's called a query into that database. Well-constructed, high-volume sites, especially e-commerce sites, will instead store or assemble the most frequently accessed information in a layer above the entire database, called a cac…

http://tucsoncitizen.com/morgue/2009/05/07/115945-mexico-s-w... Depends on what industry you work in. A Cache may look more like a giant pile of weapons as opposed to some memory in a computer.

Eh, it's not industry-dependent so much as industries have specific reductions. A cache is long-term storage. The weapons industry would expect it to be filled with weapons, and maybe some body armor. A scouting party would expect tools for equipment repair, rations, and maybe some minor comfort gear. A computer scientist would expect it to be full of bytes.

http://en.wikipedia.org/wiki/Cache

http://www.etymonline.com/index.php?term=cache&allowed_in_fr...

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#112

Post by ansible »

liber8 wrote: post #100
beat wrote: post #84

Earlier quoted context omitted.

I have high blood pressure. Thanks to insurance, I was able to spend a few months with my doctor, experimenting with medications until we found the combination that brought it down to manageable levels. Meds cost me $15 a month or so. Blood pressure medication will almost certainly add many years to my life - decades, maybe. Without insurance, I couldn't have afforded the half-dozen trips to the doctor or the meds. A…

Honest, non-snarky question for you: How is it that you could not afford a half-dozen trips to your doctor, but you could afford your medical insurance premiums? Are your premiums subsidized or free? Is the doctor terribly expensive? (I'm assuming that your doctor charges $250 or less per visit. Six visits would cost $1,500. I'm assuming those visits were spread over at least a number of months, so you could see whet…

I'm assuming that your doctor charges $250 or less per visit.

Well, that just a visit. What about lab tests? Doctor offices can do some basic stuff in-house, but for anything interesting, it has to be sent to an external lab. Depending on the tests, that could be $500 USD a pop, easy.

And the medication is expensive too. That drug may be costing the patient $15 per refill, but it definitely costs more than that in total.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#113

Post by GVIrish »

sp332 wrote: post #60
jfc wrote: post #36

> the people running HealthCare.gov had no "dashboard," no quick way for engineers to measure what was going on at the website Someone needs to disrupt the government contracts business pronto.

CGI Federal is a Canadian company that won a no-bid contract to build the site. Good luck disrupting anything; the system doesn't even pretend to be competitive and it's designed to keep incumbents in power.

To be more accurate, CGI Federal won a contract vehicle that allowed task orders to be given to them under said vehicle without doing a full bidding process. It would be like having a services contract with a company and giving them multiple projects under that contract.

Awarding work in this way isn't necessarily a mistake. But when the selection process itself is broken, you end up giving a lot of work over time to a contractor that can't do the work.

Even then, a lot of the problems with Healthcare.gov were caused by incompetence on the government side of things.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#114

Post by ryanhuff »

untog wrote: post #108
ryanhuff wrote: post #92

Earlier quoted context omitted.

I question the validity of your statement. In the US, for many uninsured, the hospital ER is their primary care outlet.

But by the time they go to the ER it's too late. People with insurance go for regular checkups that will catch things early. Those without will only go to the ER when their situation is desperate.

Perhaps you misunderstood my comment. Many uninsured in the US go to the hospital ER for minor care, such as for colds and other temporarily ailments. I suspect that these sort of cases far outnumber the visits involving serious care cases.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#115

Post by brandonb »

djKianoosh wrote: post #98
brandonb wrote: post #38

I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums. I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it: "1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're d…

I'm curious how you're breaking down the work and if you only take full time devs or are bringing in hourly contractors for different pieces. Do you think you'd have better luck finding good devs if you broke it down like that?

So far, it's been on a rotation -- folks go on leave from their jobs for 1-3 months, move out to DC/Baltimore area, and work as hard as possible while they're here. :)

It's such a high-intensity project that I think the we'll want to keep it to full-time for the time being. Otherwise, it's hard to even keep up, let alone make progress.

(As a technical point, though, even though everybody is working on this for well over 40 hrs/week, we're technically hourly subcontractors. That's just how the government works.)

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#116

Post by saraid216 »

smacktoward wrote: post #77
jfc wrote: post #36

> the people running HealthCare.gov had no "dashboard," no quick way for engineers to measure what was going on at the website Someone needs to disrupt the government contracts business pronto.

These folks are trying: http://www.dobt.co/ It's an incredibly difficult thing to do, though. Procurement practices that are enshrined in law are much harder to disrupt than those that are just bad habits that ossified over time.

There are ways to deal with things that are enshrined in law. Building a corporation is unlikely to be the right way to do it.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#117

Post by smokinjoe »

erichocean wrote: post #94
smokinjoe wrote: post #37

Earlier quoted context omitted.

PCI Compliance can be a nightmare. I don't even want to imagine HIPAA compliance for something of this magnitude.

It's not HIPPA compliant. Really, did you think the government follows its own rules? Get real.

I guess you can call me naive.

..and slightly ignorant regarding this HIPAA kerfuffle that popped up. I did some quick [hilarious] research that resulted in an issue where commented copy was being used as an indicator that there was to be no sense of privacy on the website.

source: http://www.politifact.com/truth-o-meter/statements/2013/oct/...

I'm not entirely sure that's what you're referencing or whether you're just bein' a smart aleck, but I had some fun reading up on the story. I also learned that the website healthcare.gov doesn't necessarily need as much HIPAA compliance as I had initially anticipated. To enroll, the only 'medical' type data you need to include is whether you are a smoker or not - I thought there would be much more sensitive information right off the bat.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#118

Post by mbrameld »

djKianoosh wrote: post #101
jquery wrote: post #68

Earlier quoted context omitted.

What if you'd like to help out but aren't an anointed™ one?

what do y'all mean by anointed? I've been a gov contractor for many years. It doesn't take all that much to get into this space. Basically, if you're a citizen and don't lie on applications and resumes and in interviews then there's really not any significant barrier to entry.

I think by anointed the parent means a Google or Facebook alum, or other big name from the Valley.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#119

Post by spikels »

brandonb wrote: post #38

I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums. I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it: "1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're d…

Besides the obvious issue that correlation does not imply causation what does the statement "1 in 1,000 uninsured people die each year" even mean given that the overall annual mortality rate in the US is 799.5 per 100,000[1].

Is this in addition to the background rate? If true that would be quite astounding. Or is it just rounding?

[1] http://www.cdc.gov/nchs/fastats/deaths.htm

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#120

Post by brandonb »

keithwarren wrote: post #102
brandonb wrote: post #38

I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums. I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it: "1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're d…

Aside from the rhetoric, the political opinions and people firing at you now over the 5-10K thing, I do have an actual question of substance. My assumption when things melted down so drastically, was that the key problem was integration with the various vendors. Yes we all could look at the Html itself and make assumptions about poor practices but given that you are on the inside - what was the biggest issue?

The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up.

That's what Mikey and the other Site Reliability Engineers fixed. They set up a war room with an engineer from each and every subcontractor, and the war room had three rules:

Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."

Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."

Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours."

Once you have that process working, it's the same as optimizing software: you find the current bottleneck, fix it, find the next, etc. The Time article mentions two -- the lack of DB caching and the bad ID generator. There were dozens of things like that. And still are!

Post reply on HN