CDC website built by Deloitte at a cost of $44M is abandoned due to bugs • Hacker News Board

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#291

Post by jariel » Sun, Jan 31, 2021, 5:41 AM UTC

mdorazio wrote: post #52

didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

These contracts are generally bloated, but if you've never seen how a $10M+ contract for things like this works you're probably thinking about it wrong. The actual development piece is a pretty small portion of the overall work. The bulk of it is figuring out all the requirements, wrangling legacy systems and data into some kind of shape that can either be interfaced with or imported, documentation, support (includin…

No, this is not cynical enough.

Even the part you mentioned: 'wrangling data for legacy systems' does not cost that much, it's still a small piece of the pie.

The 'bulk of it' goes to sales, BD, marketing, profits, layers of excess management, specialist consultants, accountants and lawyers.

This contract is an example of 'modern world graft'.

Healtchare.gov cost $1-2 Billion and it was also a mess until Google Engineers came in to save it.

Software is not like Construction where the costs can at least be well estimated and you don't need geniuses.

It's frankly very depressing and this kind of graft and stupidity is all over the system.

There are actually not that many people who are both: 1) technically/project competent enough to figure it out and 2) have enough gravitas on the business side to make proper bids. Partly that's the fault of government.

To be fair, it's also bad buyers who change requirements all over the place and throw projects in dispensary often for what are ultimately political reasons.

It's a real failure of leadership.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#292

Post by ethbr0 » Sun, Jan 31, 2021, 5:41 AM UTC

eigenvector wrote: post #277

nataz wrote: post #226

Earlier quoted context omitted.

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before a…

Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do? "Because we don't have the expertise to do software development." The comments in this thread, which I fully agree with by the way, just cement my belief that if gove…

This is pretty much what I've learned doing 5+ years of big enterprise consulting (though not government).

If the hardest / longest part of any project requires domain knowledge and personal relationships, then why are you outsourcing that part?

I don't care if it's not a core competency / something you can do. Make it one.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#293

Post by Cd00d » Sun, Jan 31, 2021, 5:44 AM UTC

didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

Your point stands, but a nitpick - don't forget overhead. You should roughly double engineer salary to cover overhead, so it's 146 engs at 150k (still good), or 75 engs at 300.

Either way, the deal should be possible.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#294

Post by ethbr0 » Sun, Jan 31, 2021, 5:45 AM UTC

colechristensen wrote: post #246

nataz wrote: post #226

Earlier quoted context omitted.

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before a…

But most of the rest of it in these huge contracts could just not be done. The myth is that all of the rest is necessary when only a tiny fraction really is. I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a simil…

I look at red tape / internal rules and policies as proofreading code for genetic replication.

On one hand, you limit (beneficial) mutation.

On the other hand, you limit (harmful) mutation.

An organization usually doesn't scale past a certain size without stabilizing its ability to copy itself. For better and worse.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#295

Post by pts_ » Sun, Jan 31, 2021, 5:45 AM UTC

Knew a guy who joined Deloitte. Was called star performer when he was focused on beating others down instead of the work at hand. You hire careerists, you get wall street brokers for every job, including existentially challenging ones.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#296

Post by dreamcompiler » Sun, Jan 31, 2021, 5:47 AM UTC

alasdair_ wrote: post #141

mraza007 wrote: post #109

Earlier quoted context omitted.

Wow I’m amazed like how much money can made if you are into government contracting I’m just curious to know can you get government contracts let’s say if you are running a software agency

A lot of “government contracting” is having people whose job it is to find the people that can authorize a sale and then hand-hold them through all the paperwork needed for them to actually make the sale.

This. Gov contracting is a rare, specialized skill. That's why the same big companies keep getting big contracts over and over. If you're a company without these skills you have no chance at landing one of the giant contracts. The CYA and auditing requirements are too insane to even write the proposal without a full-time staff of contracting specialists and a long track record of prior contracting.

(As a new contractor you might be able to get a small-money contract from e.g. the SBIR but those max out at roughly $100K.)

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#297

Post by JohnJamesRambo » Sun, Jan 31, 2021, 5:47 AM UTC

brundolf wrote: post #166

lmilcin wrote: post #17

I have worked with Deloitte on two projects (on clients' side, I would never willingly work for Deloitte). Generally expect them to do absolute minimum necessary to meet the contract. The code was obfuscated to ensure support contract. The people employed to write it were not interested in the quality in the slightest but were interested a lot in specifics of the contract.

Serious question: why doesn't the government have its own internal software development department at this point? One that's familiar with all the regulatory and bureaoucratic hurdles and the legacy systems, but doesn't have the incentive to grift. For the amount of money they spend on these contracts surely they could afford to pay competitive salaries?

I'm reminded of a guy that used to go on Leo Laporte's podcast and had a program or joined a program where Silicon Valley tech people join government entities to make their tech and websites better. I'm sure they took massive paycuts to do this but they felt it was their duty, like military service. This guy and others like him are true heroes and patriots to me.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#298

Post by fractallyte » Sun, Jan 31, 2021, 5:49 AM UTC

nataz wrote: post #226

mdorazio wrote: post #52

Earlier quoted context omitted.

These contracts are generally bloated, but if you've never seen how a $10M+ contract for things like this works you're probably thinking about it wrong. The actual development piece is a pretty small portion of the overall work. The bulk of it is figuring out all the requirements, wrangling legacy systems and data into some kind of shape that can either be interfaced with or imported, documentation, support (includin…

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before a…

Wait! If there was a sane set of APIs and data interchange formats that was standard across the whole system, development would be the simplest part.

It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task.

The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process.

Imagine if Deloitte had been contracted to 'invent' the Internet...

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#299

Post by ethbr0 » Sun, Jan 31, 2021, 5:50 AM UTC

belter5 wrote: post #235

nataz wrote: post #226

Earlier quoted context omitted.

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before a…

There was just an article on here last week about how software development is a learning process and programs are just artifacts of that process. Big, old, messy, and highly regulated domains will obviously be hard to learn.

Here's an example I learned on Friday:

If money is left over, apply to the next available item via (logic).

Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item.

There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second.

Using outsourced process documentation is @&$&ing useless.

Re: CDC website built by Deloitte at a cost of $44M is abandoned due to bugs

#300

Post by tvon » Sun, Jan 31, 2021, 5:50 AM UTC

didibus wrote: post #13

I just don't understand these software deals, the price is so high. At 44 million, you can hire 146 engineers each paid 300k for a full year. Trust me, you need much less to build something like this, and there's no excuse for it to suck so bad. I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?

It is not a tech problem, it is a people problem.