Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

technologyreview.com

Post reply on HN
Print view View original

341–350 of 688 posts

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#341

Post by didibus »

eigenvector wrote: post #277
nataz wrote: post #226

Earlier quoted context omitted.

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before a…

Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do? "Because we don't have the expertise to do software development." The comments in this thread, which I fully agree with by the way, just cement my belief that if gove…

Exactly, that's why I turn down companies that don't take a tech first approach, if you rely on tech, you ARE a tech company. Clearly the government doesn't see itself as a tech venture of any kind, but tech fails in silo. Tech first is the only way tech can succeed at delivering value to your business. Otherwise, go back to traditional processes, stick to SharePoint, email, excel, and manual logistics. And ya, this is true of the government as well, if it wants to modernize and leverage software innovations it'll need to start approaching things with a tech first mindset.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#342

Post by jspaetzel »

bitexploder wrote: post #336
mdorazio wrote: post #52

Earlier quoted context omitted.

These contracts are generally bloated, but if you've never seen how a $10M+ contract for things like this works you're probably thinking about it wrong. The actual development piece is a pretty small portion of the overall work. The bulk of it is figuring out all the requirements, wrangling legacy systems and data into some kind of shape that can either be interfaced with or imported, documentation, support (includin…

And then supporting it. For a decade. 44m might be bloated, but, it goes quick and it is work no one really wants to do due to how hairy and legacy it can get.

Ongoing support isn't the issue here, this is basically a greenfield project with nothing hairy to get in the way.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#343

Post by [deleted] »

afavour wrote: post #207
amscanne wrote: post #140

Earlier quoted context omitted.

> No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent. Why isn’t this just the site? I love Spartan, ugly, functional sites. That’s what the government should build, always . They should have style guidelines that are basically “no style”. You’re not competing with anyone, there’s no need to look good. Function, function, function.

One reason that comes to mind is that a lot of people browse on mobile devices. These days a site really ought to be responsive to mobile viewports and a lot of “basic bitch” government sites are not.

[deleted]

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#344

Post by arminiusreturns »

eaton wrote: post #276
eckza wrote: post #263

Earlier quoted context omitted.

> Development is literally the simplest part of the problem set. This is the single most important and insightful sentence that a lot of HN-ers are going to read today, and they don't even know it.

I'd never in a million years argue that Deloitte did an awesome job on a large government contract, but it's mind-boggling to watch the regular ritual of HN posters blithely insist that anything more over $20K and a pile of pizzas for government infrastructure is a scam.

Not only do I think thats a strawman (I don't see that base level argument given that often on hn), but even if it wasn't, the truth is there is a lot, and I mean a fuckton, of money waste at the government level on IT projects. Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed, and there are plenty of examples in even recent history that prove this point. More than just waste, the intention to fill good ol boy pockets in priority over getting the job done has significantly weakened the function of the projects, for example, the national security implications of abandoning thinthread that William Binney and Thomas Drake talk about... as just one in a myriad of examples. Virginia didn't have multiple counties become the new primary growth centers of millionares because of fair and balanced bids... and anyone pretending otherwise doesn't know how DC works.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#345

Post by WalterBright »

didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

I once hired a company to do some structural repairs. The company usually did contract work for the government. Some weeks went by. I was billed a progress payment of $5000, but the company hadn't actually accomplished anything - just reviews, several site visits, meetings, evaluations, memos, emails and constant reassignments of who was working on it.

I fired them and refused to pay their bill (they hadn't even produced a plan for how they were going to proceed). Hired another outfit, who did the entire job front to back for $5000, and got it done in a couple weeks.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#346

Post by angry_octet »

mikeho1999 wrote: post #116
didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

Full disclosure: I have had the distinct "privilege" on having worked with all aspects of large consulting firms, from working at Cap Gemini, to being a sub for a smaller company on a large gov't contract with Lockheed Martin, to actually competing (and winning) against the large companies with my own, small consulting company startup. In short, the above comment is spot on. Companies like Deloitte, Cap Gemini, PwC,…

I endorse everything you've said! Also, big consultancies have high costs (hello Booz! great lunch btw) like trade shows, political donations, lobbyists, ex-Agency/Military high muckety mucks on the payroll, etc. At least 50% of the project billings are set aside for this. Projects still need a 20% profit margin too. That might mean they're using budget staff; sometimes this means global software centres. If you think DBAs and coders in Mumbai or Warsaw might have a hard time understanding the organisational and political geography of all the States you'd be right.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#347

Post by jspaetzel »

The worst part of all of this is that the CDC didn't already have this type of tool in place, years ago, before a pandemic even hit. This is the type of basic thing they should have already in place without any new spending.

It doesn't even need to sit around unused. They should be using it every year for flu shot distribution or coordinating other types of vaccines which are normally in abundant supply.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#348

Post by sillysaurusx »

lonnyk wrote: post #303
sillysaurusx wrote: post #83

Earlier quoted context omitted.

Nope! They solved it the opposite way. Their bug tracking system consisted of individual customers who called up to report bugs. I.e. Joe Blow would call up and say "Hey, I'm supposed to see 231 shares, but I only see 219 shares." That would result in a bug report. The bug report was that Joe Blow saw that outcome. In other words, the bug report didn't contain anything about the code itself. They had an entire divisi…

How did they verify that Joe Blow was actually supposed to have 231 shares instead of 219?

I always wondered that! I assume they had a "ground truth" financial order book somewhere (which presumably was held to much higher standards of correctness) and that the support staff manually verified their balance.

But ... that logic doesn't work if you chase down the implications. And sadly I was both too shocked and too young to press my coworker for details. (He was a cool older fellow who seemed as amused with the craziness.)

Eventually I became a pentester at Matasano. During my one-year stint, I was parachuted into around 70 codebases. I got to see first-hand that Scottrade wasn't an outlier; they were the average. Most companies have similar WTFs, and the codebases are just as onerous.

The world is held together with duct-tape. That's why pentesting is so crucial.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#349

Post by bredren »

didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

This is the same old argument used against Burning Man org’s ticketing system.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#350

Post by throwhtrhwthrow »

lmilcin wrote: post #17

I have worked with Deloitte on two projects (on clients' side, I would never willingly work for Deloitte). Generally expect them to do absolute minimum necessary to meet the contract. The code was obfuscated to ensure support contract. The people employed to write it were not interested in the quality in the slightest but were interested a lot in specifics of the contract.

throwaway for obvious reasons. I've worked with Deloitte, PwC, KPMG, and other big consultancies for a diverse range of reasons (regulation audits, security reviews, staff augmentations, etc.) and the value you get for the money you're paying is pretty insane. I'd say the value you're getting is probably 20% of the price you're paying, and I'm being optimistic. You can get good value if you know the people you're hiring, if you scope your project well, and if you have clear expectations. But let me tell you, nobody knows how to do this, unless they've been on the other side.

Now, you don't always have a choice, the expertise is sometimes there, or you need one of the big four to do a prior audit before a real audit from another big four. But be prepared to have a team of OK people who will spend a lot of time producing slides and organize unproductive meetings with lots of other people (including yours) to give out a false sense of "we're making progress here". You'll have to do the PM'ing and be extremely strict with what you want from them, and you'll have to be ready to collaborate heavily. What I mean by this is: don't wait for them to ask questions, you'll have to do most of the work of onboarding them, almost like they're your coworker except that they don't have connections with the folks they need to work with, they don't have access to internal stuff, they don't know how to get there.

Post reply on HN