Live data from Hacker News

The way government does tech is outdated and risky

washingtonpost.com

81–90 of 145 posts

Re: The way government does tech is outdated and risky

#81
post #5

I would also add that the project had too many cooks in the kitchen, by some accounts. I have heard there were upwards of 50 distinct companies subcontracting on this project. I work on projects that are probably on par in terms of complexity. We typically only involve a handful of firms. And even then, coordinating them all is a challenge. I can't fathom making the process work with 50+ firms. Maybe that number was…

That doesn't seem to be the problem with healthcare.gov, or at least not in the way you suspect it might be.

The government, that is, HSS's CMS, took on the role of integrator, including integration testing (perhaps "Prime Contractor" as mentioned elsewhere). They aren't known for expertise in this (the Pentagon can do this with medium sized weapons projects, which are a rather different field anyway), and ... really screwed up:

They and those above were late with specifications and requirements, kept changing them (7 major ones in the last 10 months per the NYT), were making changes in the week before launch, and when they did a simulation test of 200 simultaneous logins just before launch the modules locked up. As did the site shortly after its midnight launch.

Oh, yeah, three days after the launch CMS panicked and proposed to fire Quality Software Services Inc. (QSSI, a unit of United Health Group) and punt their identity backend system (based on an Oracle package that's known to work), but eventually decided that would take longer than QSSI getting it to work. Who knows, but that's another sign of CMS as the integrator failing hard while distracting both QSSI and CGI Federal.

Now, maybe it ended up being too many cooks because CMS didn't provide strong oversight and coordination, but....

Re: The way government does tech is outdated and risky

#82

Federal Gov employee here. Can't speak for a project with this scope, but the procurement middlemen get into everything, far for the worse. Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We talked directly to two good vendors (good recommendations from university partners) and came up with a fine machine and 2 bids for it. Sent recommendations to procurement. Procurement put it out for bid,…

> I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent There's almost 3 million federal workers. Many more if you include people who work on government contracts. The Federal Government is by far the biggest enterprise in the US by both employees and revenue. With such a large organization, there…

'Honestly I'm not sure what a good solution would look like, but I don't think it's as simple as "trust us."'

This might be a good starting point:

http://www.amazon.com/Liars-Outliers-Enabling-Society-Thrive...

Re: The way government does tech is outdated and risky

#83
post #80

Coming from a 6+ year stint in a defense contracting, I can safely say that the issue with this approach happens well before the testing. The problem more often than not occurs at the requirements level.

Bingo. The government, which took the role of integrator, kept making requirements changes, right through the week before launch. They also did integration testing ... and of course ignored that that failed hard.

I can't see any way CGI Federal et. al. could have won.

Re: The way government does tech is outdated and risky

#84
post #76

Federal Gov employee here. Can't speak for a project with this scope, but the procurement middlemen get into everything, far for the worse. Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We talked directly to two good vendors (good recommendations from university partners) and came up with a fine machine and 2 bids for it. Sent recommendations to procurement. Procurement put it out for bid,…

The federal procurement process is one of those cases where separation of concerns hurts things. I know that things are this way to try and prevent bribery and cronyism etc. But it's clear that the pendulum has swung too far. I've been part of the procurement process a few times from the vendor side and the layers of nested black boxes makes solving procurement issues virtually impossible and once the procurement is…

"But it's clear that the pendulum has swung too far."

While I think that's likely, I'm a little bit hesitant to claim things are clear when we can't observe the alternative.

Re: The way government does tech is outdated and risky

#85
"A tank is a tank is a tank, pretty much, plus or minus a few bells and whistles."

Geeze, such amazing ignorance. If you're vaguely interested in this sort of thing, and want to learn all the process and engineering reasons the Abrams M-1 became the King of the Killing Zone, get a copy of the book by that name: http://www.amazon.com/King-Killing-Zone-Orr-Kelly/dp/0393026...

Written by someone who initially expected to castigate it due to early (mis)reported teething problems (e.g. the whole "it throws tracks (more than other tanks)" was due to a proving ground's faulty tension meter), he got completely sold on the tank which has since totally proven its worth.

Lots of fun stuff, from their modeling everything with strong constraints like weight (i.e. what bridges can it cross), e.g. they didn't want to provide a heavy M2 .50 BMG but the tankers demanded it. To the successful development team's leader, a grizzled Chrysler car exec who drove them crazy with "that doesn't look good" sorts of complaints.

Which often turned out to be a boon (ignoring that weapons should look good so their users feel good about them, which the M-1 delivers on). Said it was too high in an ugly way, so they figured out how to shave a foot off, which is very important for the European theater (not so good for deserts). Didn't like how the armor skirts didn't extend all the way to the back. So they gave in (I'm sure the modeling said it was only a minor net loss) ... and found that made a critial difference in keeping cruft thrown up by the tracks out of its turbine engine.

Very much an iterative process, in a domain where you truly "bend metal" to get things done.

So take the author's words with a big grain of salt, she's woefully ignorant of a huge domain in which we've been building for a very long time the world's most sophisticated artifacts, and learning how to, and how not to do it ... with stakes no less than national survival. Digital computers used for IT are a very recent development as these things go.

Re: The way government does tech is outdated and risky

#86
post #75

The US (and rest of world) should take a leaf out of the UK's recent initiative: GDS (Government Digital Services) http://digital.cabinetoffice.gov.uk/ Aside from creating https://www.gov.uk/ which laid down a lot of principles on how to fulfil a government contract (as well as the foundations of what goverment websites should look like and how they should be developed), GDS is also looking at the problem of procurem…

This week a UK parliamentary watchdog described a failed National Health Service patient IT programme – the cost of which has spiralled to £9.8bn – as “one of the worst and most expensive contracting fiascos in the history of the public sector”. Earlier this month the Department for Work and Pensions admitted that it had written off £34m of IT costs, incurred in an attempt to overhaul how social security benefits are…

All outsourced.

GDS is trying to in-source a lot of work that should be controlled by a central publishing and transaction design group, and contracting out the relatively boring task of following their rules. Good idea, as long as their architecture/design/program management review boards are well staffed and motivated.

I don't think they will stop these large failures. And many of these large failures do have clear up-front requirements that cannot be changed. Iterative waterfall is not so terrible.

Re: The way government does tech is outdated and risky

#87
post #86

Earlier quoted context omitted.

This week a UK parliamentary watchdog described a failed National Health Service patient IT programme – the cost of which has spiralled to £9.8bn – as “one of the worst and most expensive contracting fiascos in the history of the public sector”. Earlier this month the Department for Work and Pensions admitted that it had written off £34m of IT costs, incurred in an attempt to overhaul how social security benefits are…

All outsourced. GDS is trying to in-source a lot of work that should be controlled by a central publishing and transaction design group, and contracting out the relatively boring task of following their rules. Good idea, as long as their architecture/design/program management review boards are well staffed and motivated. I don't think they will stop these large failures. And many of these large failures do have clear…

Funny thing is, as far as we can tell this is how that project was done, minus the contract size sorts of splits. The government didn't hire a integrator, HHS's CMS took on that responsibility including integration testing.

Of course the minor fly in the ointment is that CMS didn't even vaguely have the expertise to pull this off; the Pentagon can do this for medium sizes weapons projects (which are a rather different field), but no one else in the US government has been said to have it.

Re: The way government does tech is outdated and risky

#88
post #78

Earlier quoted context omitted.

One case I've heard is when you don't have any changing requirements and the domain is well known. E.g. engine control software.

If the requirements don't change and the domain really is well-known, why are you writing new software? Why isn't there pre-existing software you can reuse?

Not in my backyard syndrome.

Re: The way government does tech is outdated and risky

#89
post #46

That diagram for the "waterfall" approach that they yanked from Wikipedia is a complete straw-man representation. It's nonsense. Here is the actual, original source for the Waterfall approach, first published in 1970: http://leadinganswers.typepad.com/leading_answers/files/orig... If people would just bother to scroll past the first couple of pages, they will notice that the approach already includes some iteration c…

"What often happens when you have these big requirements up front, is the people who are specifying the product are afraid of not getting all their ideas in, so they overscope the project. And then the development team is on the hook for delivering everything, not just the essential elements." Isn't this the essence of the critique, though? Its not the lack of iteration, but the logic of the spec-formation. To put th…

Here here!

It would be lovely if vendors would bid based on their experience, and be compensated for it, at a time and materials basis. The more experience and better you prove to be, the higher we're willing to pay for a better outcome, sooner.

Except that world is rife with bait and switch. And writing the code that delivers the spec is just a small part of the picture. Companies are terrible at specifying the non-functionals, delivery process expectations, operational requirements, supportability requirements, etc. When they do get it right, the costs go up, because many places quote without any concept of these things.

The sad reality is that the people who get asked to quote for these things in the software world are generally clueless about the actual domain, out of touch, and wildly wrong most of the time. And the people that specify these things are often barely any better. And lets face it, software developers are also terrible at quoting times to do a task, though that can be mitigated with a lot of experience of that task and the code base to do it on.

Re: The way government does tech is outdated and risky

#90
post #73
post #3

For the MIT alums out there, I remember a 6.170 exam that had the question: "When is it appropriate to use the waterfall model of development?" The answer was any time you are developing software for the government! The professor specifically mentioned it in lecture once, so that alone was enough for full credit on the question (other reasonable answers were fine too). Later I TA'ed the class twice and made sure to e…

Interesting fact - the original design of "waterfall" isn't what we perceive as "waterfall" today: http://leadinganswers.typepad.com/leading_answers/files/orig... My theory - agile/iterative development rarely gets sold because we continue to believe we aren't susceptible to planning fallacy.

What do you perceive as waterfall? That's exactly the way we learned it in school...
Post reply on HN