Live data from Hacker News

Bank of America's CEO says it's saved $2B per year by building its own cloud

businessinsider.com

151–160 of 321 posts

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#151

I work for banks, and I can tell you that when you are doing devops with their "own cloud", you are miles away from a real cloud experience : no os choice, no hardware choice, slow provisioning, no access to repo, low and inconstant virtual disk (EBS) speed... I guess that you get what your paied for, and maybe the 2B saved on cloud are spend on IT service that suffer from such a poor own cloud experience. If they ha…

50,000 engineers just to run a bank? I don't think so.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#152
post #132

Earlier quoted context omitted.

How do you spend that much on software and have such an abysmal usability and security story? I don't think there is anything technically difficult about the consumer software they offer, namely https://www.bankofamerica.com/ .

I think you would find that their security is better than you think. Otherwise they'd be hemorrhaging money left right and center to North Korea and the likes. As for usability, probably a degree of incompetence, mixed with design-by-committee and legacy. Edit: It's worth pointing out that banks usually don't gain or lose customers based on their UX, so it's not something that the business optimizes much. I don't thi…

> I think you would find that their security is better than you think.

You can get in to my account by verbally relaying my grandfather's first name over the phone. You can open a bank account with a SSN and no photo id. What security? Their "security" is a fraud department, much like our credit card industry.

> Otherwise they'd be hemorrhaging money left right and center to North Korea and the likes.

This is not how transactions work.

> Edit: It's worth pointing out that banks usually don't gain or lose customers based on their UX, so it's not something that the business optimizes much.

All banks offer the same shitty experience. What does differentiate them if not their software? They offer literally nothing my local credit union doesn't offer.

> I don't think there is anything technically difficult about almost any of the consumer software facebook offers, except scale.

No argument here, but facebook at least manages to hire designers and not impose weird non-sensical patterns of auth, like "Look for this image when you log in".

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#153

> The bank, which has a $10 billion annual tech budget Am I the only one completely befuddled by this number? What the fuck are they doing with these money and 200k servers? These are facebook numbers. For a bank. What?

The sell-side investment bank alone is a huge consumer of this tech budget. 13 exchanges x 3000+ securities x 2 quotes for each security x 23,400 seconds in the trading day (with the very very very generous assumption that quotes only change once per second) That’s the inbound data alone, now consider all the algorithms they have to run on it, the models, the strategies, etc. And then to each of their several hundred…

And how much data visualization do they do? Software devs vastly underestimate the gravity of the data visualization situation.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#154

I think a strong argument can be made either way. There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation. This isn't simply about having fewer s…

I worked on the internal BOA cloud. My team wrote software to procure VM's via a website, versus the older (still current) way of doing it manually.

There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility.

On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#155
post #138

Earlier quoted context omitted.

How do you spend that much on software and have such an abysmal usability and security story? I don't think there is anything technically difficult about the consumer software they offer, namely https://www.bankofamerica.com/ .

Why do you say they have bad security? I've talked with info sec employees working in their Charlotte office, and have not been given an indication that they are slacking.

Their security questions are a massive security hole. Credit cards don't require a PIN. I don't see much indication that it is difficult to steal from people.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#156

Many large companies save money by having their own cloud, but those clouds sometimes really suck compared to AWS. I know of one very large company where any request for a change in their cloud infrastructure always required a minimum two week advanced notice. In AWS such a change is just a mouse click away, and could be done in seconds. AWS also has a really amazing integration of a large variety of services which i…

That's not a comparison of external cloud vs. self hosted though. It is a comparison of different operating models with respect to change control. Either one can exist in both external vs. self-hosted infrastructure.

I've been in places where the engineering team plugs in an old PC under a desk somewhere, gives it a public IP address of there's the production server. That's the self-hosted equivalent of the engineering team having full AWS access and any change is a mouse click away.

I've also experienced places where any change does take a week or two of approvals even though it is hosted on public cloud.

There is a time and place for all approaches. What works best with a three person startup putting up a MVP is quite different from what is best in a very large corporation operating in a regulated environment.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#157
post #81

Earlier quoted context omitted.

Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.

Maybe it’s because wages are a lot lower in Europe? Better to outsource to the cloud than hire more people if people are expensive.

IDK where you get the 'wages are a lot lower in Europe' thing. There are lower cost and higher cost places, but people are expensive and talent is mobile.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#158
post #24

I'm not really sure why this should be a surprise. Just as big companies use a mixture of owned and rented real estate for various reasons, the same is true of other large expenses. If you have a large, predictable, core workload it makes sense to bring it in house, and use the elastic (rented) resources for unpredictable stuff (e.g. new efforts). It's not like Amazon doesn't know this too (e.g. they try to make the…

I don’t think proprietary is offered to make transition harder. In a lot of cases they simply offer a better option.

Well that's definitely what they will tell you!

There are plenty of reasons to choose a point in the space denominated by the axes make/adapt-existing X insource/outsource X proprietary/open . None are zero cost unless you really think your platform is good enough that enough others will write for it without you doing anything.

It's possible that you would chose to write your own because nothing in the market is even close to good. It's possible that you would open source it in the hopes that it becomes standard and that others will add new features that you will benefit from too (this is not a zero-cost option either). It's possible that you would even keep it proprietary because that's cheaper. But given the way Amazon operates it's pretty clear that they offer their own proprietary services to lower the barriers to adoption ("simply offer a better option") and raise the barriers to switching.

I don't see anything wrong with this strategy legally or morally, though I mostly avoid amazon proprietary tools because the risk of locking is one I'm not willing to embrace. There are of course situations where they aren't a problem (e.g. a tool with an anticipated finite lifetime: the cost of lock in in that case is essentially nil).

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#159
post #18

If you know your traffic well, private cloud saves a lot of money. Few years back i bet hybrid clouds will take over and for some reason it did not happen. I still believe hybrid cloud is the solution for mid size and up companies. You definitely need cloud provides for handling traffic spikes.

Nobody ever got fired for using AWS.

IT decisions that get people fired are usually not well known to the public anyways.

Re: Bank of America's CEO says it's saved $2B per year by building its own cloud

#160
post #114

“Right now, the bank estimates its private cloud is 25 to 30% cheaper than public providers, though it also recognizes that probably won't last forever.”

After a meetup this spring, someone talked my ear off about how Dell is highly motivated to have a private cloud solution that works for people.

Having Dell and Amazon in a bidding war over your next project is probably the best world you can be in.

For my money, you should run one data center in the same location with most of your tech talent, and a second one geographically distant, and regionally load balanced.

But we typically don’t write our software for this, and you can’t get the business to do a rewrite until they see how stupid expensive a lift and shift ends up being.

I think these narratives about saving money are typically covering up a story of how much was squandered starting five quarter ago...

Post reply on HN