Live data from Hacker News

Bank of America's CEO says it's saved $2B per year by building its own cloud

businessinsider.com

211–220 of 321 posts

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#211
post #81

Earlier quoted context omitted.

Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.

Maybe it’s because wages are a lot lower in Europe? Better to outsource to the cloud than hire more people if people are expensive.

More likely about trust/security/privacy issues.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#212

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

does that account for the zero capex?

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#213

That implies their business would have data center costs beyond most tech companies (Lyft for example is spending $300 million with AWS over three years, Snap is spending $3 billion over five years). Something doesn't add up here.

Tech companies don't necessarily spend more on computing resources than non-tech companies. The article says they had 200,000 servers in 60 datacenters. That is certainly more than most tech companies, and is not surprising. A large bank probably has more computing needs than someone like Lyft.

200,000 servers isn't actually that many. 200,000 c5.xlarge's at on demand rates (which nobody with 200,000 servers pays) will cost you about $300 million a year. So they are realizing savings at about an order of magnitude higher than you'd expect cost wise.

I used to work in banking... their compute needs are actually quite basic.

I think what we're really seeing here is some creative accounting around how operations and costs are accounted for. They've got to be talking about costs of managing and operating their cloud as well, and most of those costs would be in terms of training & workforce retooling.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#214
post #34

I think a strong argument can be made either way. There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation. This isn't simply about having fewer s…

> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I think some companies, like BOA, look at this as the safer alternative. We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world con…

BOA also didn't (doesn't) really trust the public cloud. During several all hands meetings in the org, the inevitable question of "when are we gonna start using the public cloud" came up and the leaders would always say "we aren't because it isn't secure enough" or "we'd rather maintain our current governance model over what our devs build".

To your point, we need only point to recent events to show the universal customer pain that can be caused by a bad code push at Azure or AWS.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#215
post #67

That's almost as much as they make on overdraft fees!

That's unfair - they were only fined $67 million for fraudulent overdraft fees! Though I have to wonder how much they're currently making by inducing every one of their customers to write fraudulent checks.

Though there is something comforting about a predictably evil company though. Much more straightforward than "do no evil" while building a surveillance machine that would make a Stasi officer blush, or "this helps our users" while hindering Hong Kong's protests.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#216

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

> It will continue to rise because it offers flexibility ... The cloud is more expensive for large companies, for most workloads. The way I read this is that the operating costs are higher in the cloud, but presumably dwarfed by recouped opportunity costs. This makes sense to me, given my limited experience working in large companies where every interaction with IT involved a 2-6 week turnaround time. Need a VM? Fill…

A private cloud can fix that turnaround time just as easily.

The true opportunity cost in large corporations is from, IMHO, been the general attitude of "we can't do foo new thing because that's [not best practice, not (barely) supported by some other large corporation, too cheap, too big of a change, etc]."

The ability to create extremely flexible workloads in minutes means nothing when security has to approve sneezes and containers are only something you'd find in a fridge.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#217
post #34

Earlier quoted context omitted.

> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I think some companies, like BOA, look at this as the safer alternative. We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world con…

BOA also didn't (doesn't) really trust the public cloud. During several all hands meetings in the org, the inevitable question of "when are we gonna start using the public cloud" came up and the leaders would always say "we aren't because it isn't secure enough" or "we'd rather maintain our current governance model over what our devs build". To your point, we need only point to recent events to show the universal cus…

A factor that is easy for developers to overlook is that the legal system still cares who owns the hardware. So if all your stuff is at Amazon, a warrant can be served to Amazon and they will let the cops into your stuff. No contract can keep that from happening.

If a company (or person) really wants complete visibility and control into their relationship with law enforcement, they have to host their own hardware... ideally, their own data center.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#218

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

I used to work at a very large and entrenched publishing-cum-tech company, too. Sounds like they were different companies, but very much the same rationale for migrating to a well-known cloud service provider. The ossified and entrenched IT department was becoming one of the biggest challenges to quickly reacting to market challenges and opportunities, and was long overdue for a few boots planted firmly to the rear end.

The cloud was probably always more expensive if you only considered the cost of computers. But it was about so much more than that. Mostly, it was about creating a hostile environment for the BOFHs who were creating a hostile environment for everyone else.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#219
post #156

Many large companies save money by having their own cloud, but those clouds sometimes really suck compared to AWS. I know of one very large company where any request for a change in their cloud infrastructure always required a minimum two week advanced notice. In AWS such a change is just a mouse click away, and could be done in seconds. AWS also has a really amazing integration of a large variety of services which i…

That's not a comparison of external cloud vs. self hosted though. It is a comparison of different operating models with respect to change control. Either one can exist in both external vs. self-hosted infrastructure. I've been in places where the engineering team plugs in an old PC under a desk somewhere, gives it a public IP address of there's the production server. That's the self-hosted equivalent of the engineeri…

Change control and service delivery time-frames are two different things. When I left BOA in 2017, you could get most production changes implemented with only two days of lead time from a formal change control perspective. But, requesting a new Virtual IP on a load-balancer could easily take two weeks, just for every layer of bureaucracy to wet its beak. And it was impossible to request something so basic without an online service request, and then follow-up emails because the standard service offerings left all kinds of details undetermined and no structured way to provide the information.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#220

I used to work at a very large financial publishing company. I joined just at the point where they were moving from VMware to AWS. It was never a cost saving exercise. It was a way to completely change the culture of the IT department. As they were one of the very first news websites, they had at the time ~18 years of cruft to deal with. The drive to the cloud was about flexibility, and if we are honest, shaking the…

> the opex was close to £3mil in AWS bills alone What was the opex before cloud?

Data centers and the contracting companies’ overhead of managing them. If they’re not that great at it, you lose a lot in terms of availability, business agility, etc. Like, imagine using Softlayer as your provider except even worse. You have to manage ticketing systems for these folks specifically, you need to handle leases for colocation, ad infinitum when you have to manage an enterprise class datacenter and even then it doesn’t mean you get enterprise results.
Post reply on HN