Live data from Hacker News

Bank of America's CEO says it's saved $2B per year by building its own cloud

businessinsider.com

201–210 of 321 posts

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#201

Earlier quoted context omitted.

s/employee/engineer/2 (Note the 2.)

What tool is this for? Sed? I've copy pasted it before, but i think you just made the syntax click

Yep, sed supports that syntax (substitute). echo 'There are ~250k employees at JP Morgan, not employees.' | sed 's/employees/engineers/2'

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#202

Can anyone who has worked on Bank of America comment on if any of the things they build actually worked?

This must be a jab, but obviously yes. Last year they had $28 billion in net income. With that kind of money you can hire the best in the world to build literally whatever you want. They could go build a space shuttle if they felt like it - probably without much trouble. In the unlikely situation that it didn't work, they'd pay someone else to fix it or re-do it. The least likely (albeit not unprecedented) end situat…

[deleted]

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#203

I think a strong argument can be made either way. There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation. This isn't simply about having fewer s…

> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff. In reality it is quite the…

I don't necessarily think you're wrong, but I think you're missing the point.

Situation: some manager is tasked with updating some antiquated COBOL monstrosity. They're given twelve peanuts, some shoe string, bubble gum and a little bit of duct tape. The experienced staff have all been aged out, quit out of frustration or laid off and replaced with fresh college grads for half the price which was a great deal and significantly reduced costs for several consecutive quarters, earning a well deserved healthy bonus. Unfortunately reliability of the system has been suffering and the young engineering team unanimously agrees the old system should be replaced.

Option 1: do it in house with the young team of rockstars the manager has been bragging to management about. Unfortunately, the manager secretly has reservations and has doubts about the team's experience, let alone skills. It's too risky. Not too risky to the company who can easily absorb the loss and move on, but too risky for the middle manager who will look bad and possibly get fired.

Option 2: outsource the entire thing to IBM or Oracle. Tempting, but not in the budget. Also, significant parts of the old system are on IBM anyway and the cost of hiring IBM people to fly out has been increasingly expensive after the latest round off layoffs saw Scott put out to pasture. No, it's too risky. Not risky to the company who can have uptime guarantees in the contract, but risky to the manager who will look bad if he comes in over budget.

Option 3: The Cloud. It's the 'in' thing. FAANG are all all in on it, and if it works for them it could work for us. Much of the new team used AWS for their web dev class in college, so the skills are fresh. There's very little risk to it. The staffing requirements will be dramatically reduced so costs will drop. The manager will be able to deliver an awesome demo next quarter and the S3 instance they'll host it on will only cost one or two of their peanuts. The only risk is if the manager doesn't get promoted before the problems start, which shouldn't be a problem because AWS has such great onboarding programs. It is slightly more risky for the company though, who may find themselves no longer in control of critical infrastructure.

Risk for the company isn't the same as risk for a mid level decision maker. Small fuckups carry the same penalty as huge fuckups. (you get fired) From my fake story, doing it in house has a relatively high chance of being a minor fuckup, and AWS has a relatively low chance of being a major fuckup. The manager is going to choose the low probability risk, even though the low impact risk is the company's best interest.

And let's be honest... a remarkably large fraction of small IT shops are really bad at certain basic competencies, notably backup reliability and timely application of updates. AWS is pretty good at sysadmin and accounting 101, even if it stumbles at 201.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#204
I have a question: I'm doing an ambitious Web site startup; current status, rushing to alpha test; and from this thread and more I wonder about servers and our server farm and in-house network, that is, for the options here, in-house, co-location, AWS, Azure, etc. I'm using Microsoft's software, Windows, .NET, SQL Server, etc.

Q. For many of the reasons given in this thread, etc., I'm leaning to having our own in-house servers, server farm, etc. For that, is there a source of information I will be able to use, say, as consulting, depending on the issue, an hour, day, week, month at a time, to get us past the chuckholes in the road on the usual work -- system and network planning, installation, configuration, monitoring, diagnosis, correction, etc.? E.g., can I just call Microsoft for such issues, CloudFlare, VMWare, Cisco, etc.? Assume that money to pay for the products and consulting will not be a problem.

If the broad answer is "Yes", then that will take a lot of entries off my TODO list and let me sleep better.

Thanks.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#205

Earlier quoted context omitted.

Maybe it’s because wages are a lot lower in Europe? Better to outsource to the cloud than hire more people if people are expensive.

IDK where you get the 'wages are a lot lower in Europe' thing. There are lower cost and higher cost places, but people are expensive and talent is mobile.

They 100% are. They're not even close.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#206

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

> It will continue to rise because it offers flexibility ... The cloud is more expensive for large companies, for most workloads.

The way I read this is that the operating costs are higher in the cloud, but presumably dwarfed by recouped opportunity costs. This makes sense to me, given my limited experience working in large companies where every interaction with IT involved a 2-6 week turnaround time. Need a VM? Fill out a bunch of paperwork and then wait a month. We missed a lot of opportunities that way.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#207
post #78

Looks like $2B of savings are from reducing the number of servers (200,000 servers earlier to 70,000 now). That has nothing to do with the advantages of a private cloud over public, etc. They were just over-deployed and by reducing the number of servers they would have enjoyed similar savings no matter where they were running them. In fact they could have saved even more, and earlier, if they had already been in the…

Is $15,000 per server normal?

I would imagine switches, firewalls, UPS, temp control, fixtures, and facility security are also significant cost factors.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#208
post #34

Earlier quoted context omitted.

> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I think some companies, like BOA, look at this as the safer alternative. We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world con…

> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of c…

Hopefully you'd think they compared those costs and still found the in-house option would be cheaper in the long run. Who knows, though, or who knows how accurate their estimates were.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#209

Earlier quoted context omitted.

> This is a myth directly from cloud providers marketing campaigns I agree the cloud providers like this idea, but I'm not sure I agree it's a myth. Businesses need to choose what competencies to focus on. For nearly every other area, businesses rely on suppliers. Auto manufacturers use machines built by other businesses and parts built by other businesses. They also obviously do some amount of in-house part and tool…

> Auto manufacturers use machines built by other businesses and parts built by other businesses. Yeah but they generally aren't renting those machines. Renting critical pieces of business infrastructure that aren't easily replaceable is opening yourself up to rent seeking on the part of your supplier. Remember the Oracle business model--lock people in and then keep raising the price just below what it would cost to r…

Why is it better to be locked into your own company’s IT department rather than AWS? I would argue AWS gives a better ROI than 95% of corporate IT departments, and generally less hostile to boot.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#210

Earlier quoted context omitted.

IDK where you get the 'wages are a lot lower in Europe' thing. There are lower cost and higher cost places, but people are expensive and talent is mobile.

I don’t know, in the Bay Area 150k is entry level and big companies pay 250k or much higher to senior engineers. Whenever I hear about pay in Europe it’s a fraction of that. It may be worth it for a different lifestyle, I’m not making that argument. Regardless, from the company POV it must lead to different choices about buy vs. build.

Bay area or NYC sure, that's like being in London, but what about the Midwest, or any of the places where you can easily set up an IT shop without paying a premium for space?
Post reply on HN