Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov • Hacker News Board

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#121

Post by keithwarren » Thu, Feb 27, 2014, 10:31 PM UTC

brandonb wrote: post #120

keithwarren wrote: post #102

Earlier quoted context omitted.

Aside from the rhetoric, the political opinions and people firing at you now over the 5-10K thing, I do have an actual question of substance. My assumption when things melted down so drastically, was that the key problem was integration with the various vendors. Yes we all could look at the Html itself and make assumptions about poor practices but given that you are on the inside - what was the biggest issue?

The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up. That's what Mikey and the other Site Reliability Engineers fixed. They set up a war ro…

What is the biggest issue right now architecturally?

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#122

Post by mildtrepidation » Thu, Feb 27, 2014, 10:41 PM UTC

jfc wrote: post #36

> the people running HealthCare.gov had no "dashboard," no quick way for engineers to measure what was going on at the website Someone needs to disrupt the government contracts business pronto.

This is not the "business that happens in a vacuum or at least on the level of normal capitalism" "business" or industry you seem to think it is.

I know people who are involved in this. Just being able to bid takes decades and a network you either can't buy or need to be very rich to buy into. It takes either already being there or finding excellent bonuses (e.g. the kind of benefits that come with setting up your business in Detroit). And then you have to compete with the entrenched interests.

This is, or at least can be, far down the ladder from Lockheed-Martin. But it's the same kind of environment: If you're already there, it's easy money. If you're not, it almost certainly out of your league. If you know how to play the game, you still need to know the right people. If you do know the right people, you need to grease their palms. And once you've done that, you're part of the problem.

"Someone needs to disrupt x" is a magically simplistic and meaningless point of view.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#123

Post by jbooth » Thu, Feb 27, 2014, 10:42 PM UTC

hga wrote: post #97

jbooth wrote: post #51

Earlier quoted context omitted.

You're not wrong, but anyone who actually has an office in the physical whitehouse, or anyone within, say, 5 layers of the President, actually shouldn't be concerned with the implementation of the technology. They should be focused on getting money and resources to the people who do the tech and running interference for them to help prevent them from being sabotaged by political opponents. In between their other mill…

" They should be focused on getting money and resources to the people who do the tech and running interference for them to help prevent them from being sabotaged by political opponents. " Can you suggest some effective mechanisms by which "political opponents" have been able to "sabotage" their efforts? Do these people get depressed when they hear an eeeevil Republican saying what they're doing is wrong, or even evil…

Shutting down the government for a few weeks probably complicated some things. Can always call non-stop hearings and bring people in to talk instead of work, as well.

What, you think nobody plays hardball in DC?

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#124

Post by aryastark » Thu, Feb 27, 2014, 10:44 PM UTC

aristus wrote: post #29

dannyr wrote: post #23

I heard that several people from YC startups and Google are helping out. This is one of the reasons why we're seeing a lot of improvements to the Federal Exchange site.

A friend of mine, from Facebook's networking team, also spent a month there. He came back with a thousand-yard stare.

there is apparently a generational gap here. There are people that have read The Mythical Man-Month, and then there are people that go and re-live it.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#125

Post by brandonb » Thu, Feb 27, 2014, 10:45 PM UTC

keithwarren wrote: post #121

brandonb wrote: post #120

Earlier quoted context omitted.

The biggest general problem is that people thought of shipping the site the same way they thought of shipping an aircraft carrier -- you write the code, hand it over, and you're done. It wasn't treated as a running service. So, for example, when the site went down, there wasn't a group of people responsible for bringing it back up. That's what Mikey and the other Site Reliability Engineers fixed. They set up a war ro…

What is the biggest issue right now architecturally?

Complexity. We have 10x more code, 10x more components, and 10x more layers than we need. If the initial architecture had been dirt-simple, I don't think the site would have had so much trouble scaling or staying up. But, of course, removing complexity without breaking things takes longer than adding it in the first place.

The other big problem is operations — many steps are done manually which should be done with tools like chef or puppet. When you have a lot of manual steps in your deploy process, it makes the whole system harder to scale, test, modify, and keep running. "Devops" has become a buzzword but it's definitely needed here.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#126

Post by hga » Thu, Feb 27, 2014, 10:55 PM UTC

beat wrote: post #110

liber8 wrote: post #100

Earlier quoted context omitted.

Honest, non-snarky question for you: How is it that you could not afford a half-dozen trips to your doctor, but you could afford your medical insurance premiums? Are your premiums subsidized or free? Is the doctor terribly expensive? (I'm assuming that your doctor charges $250 or less per visit. Six visits would cost $1,500. I'm assuming those visits were spread over at least a number of months, so you could see whet…

Honest, non-snarky answer... to be pedantic, for me it's not a question of affordability. I'm a software professional with a very good income, and I could do it on my own. I have a friend a few months younger than me. She makes less than $15k/year, and has two children living at home. Under those financial circumstances, she has totally paid off her home, which should tell you how frugal and responsible she is. She h…

"Medical costs are the leading cause of bankruptcy in the US."

In a study by Elizabeth Warren that scored any bankruptcy which included medical bills in the discharged debts as their "being the cause" of it, no matter what the fraction they were.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#127

Post by watwut » Thu, Feb 27, 2014, 10:56 PM UTC

sp332 wrote: post #59

pella wrote: post #10

Dickerson rules: "Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame." Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accur…

Rule 3 is what managers are for. The engineers will want to work on cool features or challenging problems, not the most urgent ones.

Only if you employ bad or only junior engineers. If you can not deal with priorities, then you have no right to call yourself senior. Especially when it comes to urgent.

Good engineers are able to finish work even if you do not babysit them.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#128

Post by hga » Thu, Feb 27, 2014, 10:57 PM UTC

jbooth wrote: post #123

hga wrote: post #97

Earlier quoted context omitted.

" They should be focused on getting money and resources to the people who do the tech and running interference for them to help prevent them from being sabotaged by political opponents. " Can you suggest some effective mechanisms by which "political opponents" have been able to "sabotage" their efforts? Do these people get depressed when they hear an eeeevil Republican saying what they're doing is wrong, or even evil…

Shutting down the government for a few weeks probably complicated some things. Can always call non-stop hearings and bring people in to talk instead of work, as well. What, you think nobody plays hardball in DC?

For the former, we've heard that nobody important stopped working during the "shutdown". The latter was entirely a hypothetical ... before the disaster. After, I don't recall anyone truly important being detained in hearings for long (more than a day? with presumably 1-2 days of prep time at most before?), and the Administration has had no compunctions about not sending people to Congressional hearings if they don't want to.

"What, you think nobody plays hardball in DC?"

National level Republicans as of late have been playing badminton at best.

That you cannot cite "some effective mechanisms" after making such a broad claim suggests to me that you're approaching this discussion as a political, not a technological, one.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#129

Post by samstave » Thu, Feb 27, 2014, 10:58 PM UTC

brandonb wrote: post #38

I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums. I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it: "1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're d…

The least of my worries is government enforced purchasing of over-priced, broken healthcare.

How about sending to prison those that paid nearly 1BILLION for this site as well as the bankers and defense industry gluttons who stole trillions from us in the last ten years?

If I want to make my government modern - I'd put in place a system of accountability.

Re: Obama's Trauma Team: Inside the Nightmare Launch of HealthCare.Gov

#130

Post by cynusx » Thu, Feb 27, 2014, 11:00 PM UTC

jowiar wrote: post #86

jbooth wrote: post #51

Earlier quoted context omitted.

You're not wrong, but anyone who actually has an office in the physical whitehouse, or anyone within, say, 5 layers of the President, actually shouldn't be concerned with the implementation of the technology. They should be focused on getting money and resources to the people who do the tech and running interference for them to help prevent them from being sabotaged by political opponents. In between their other mill…

> That said, obviously they should have brought in the campaign website team... That's called cronyism, and, for the most part, it's either against the law, or will get you destroyed in the media. One of the biggest challenges in government is that, for those of us in startup-land, the process goes something like: "I need to hire someone, I'll call my friend who I know does good work". You can't really do that in the…

Procurement process design is not only a problem in public bodies, there is a lot of corruption in private companies as well where a procurement person would strike a deal with a sales person. This deal can range from "a nice watch" to "I like a swimming pool", depending on the contract value.

That said, I also think that the current procurement methods are broken and that a less rigorous procurement process would eliminate contractors that specialise in government procurement.

One way would be to implement a randomised audit where the deal would be re-examined and the procurement officer would have to defend every choice he made. if 5% of the deals were audited then bad actors would be caught very rapidly.

An other approach would be to adapt the jury-trial approach to government procurement and have them review every procurement contract.

Third approach would be to combine this to reduce the workload.