How Canva saves Amazon S3 costs • Hacker News Board

Re: How Canva saves Amazon S3 costs

#131

Post by alecco » Mon, Jun 19, 2023, 6:33 PM UTC

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

And now it's VC-fashionable to have S3 as "object store" for "serverless" databases. "offload to S3!"

Re: How Canva saves Amazon S3 costs

#132

Post by littlestymaar » Mon, Jun 19, 2023, 6:35 PM UTC

jabart wrote: post #78

littlestymaar wrote: post #38

Earlier quoted context omitted.

I keep seeing that argument, but you, like the others, don't provide the figures you claim you got from your calculation. Also, it seams that it's 230PB in total , not per month, which fits in an apartment-sized server room (you want to have several of such places, but it's not that big).

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. htt…

This is a marketing pitch for a cloud storage company and it shows:

> - System hardware: $500,000

For one PB? Gold plated HDDs and cases I guess…

> Assume 15% annual system maintenance over five years

Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s

According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to expect over 6 years of consecutive use!

> Assume the space needed to store 1PB is between 25%-50% of a standard rack (42U)

You can fit almost 10PB in a rack though, so that's another figure that they've inflated.

> Assume personnel costs to manage a 1PB system are 0.5 IT/storage admin FTE

This one is probably true (and it's even an underestimation) if you have only 1PB of data, but it's not going to scale linearly: you're never going to need 130 FTE for 260PB.

Overall, if a supplier (who's going to use the same kind of off-the-shelf tech as you would) claims that his price is 4 times cheaper that what it would cost you to run it by yourself, he's probably just lying to sell you stuff, just saying…

Re: How Canva saves Amazon S3 costs

#133

Post by jedberg » Mon, Jun 19, 2023, 6:38 PM UTC

andrewstuart wrote: post #20

onion2k wrote: post #6

Earlier quoted context omitted.

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops This is the traditional “sell” for cloud computing and I don’t buy it at all. The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems. The clouds sales pitch is the if you go cloud then you…

When I moved reddit from on-prem to the cloud, I cut our costs by 27%. It was the same number of people managing both infrastructures, but once we moved to the cloud, I no longer spent most of my time imaging machines and driving them to a datacenter to rack and stack them. Instead I spent my time coding ways to manage machines via API, so that when I needed to double our infrastructure, I just ran a script.

Cloud makes a ton of sense for high growth companies. If your infrastructure is mostly static, that's when it makes sense to go to a datacenter. Or if you have one very specific use case, like Dropbox.

Re: How Canva saves Amazon S3 costs

#134

Post by jabart » Mon, Jun 19, 2023, 7:01 PM UTC

littlestymaar wrote: post #132

jabart wrote: post #78

Earlier quoted context omitted.

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. htt…

This is a marketing pitch for a cloud storage company and it shows: > - System hardware: $500,000 For one PB? Gold plated HDDs and cases I guess… > Assume 15% annual system maintenance over five years Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to…

Can you fit 10pb in a rack, sure if you want it all flash. You also need double that for mirroring, or quadruple that for raid 10. Double that if you want a mirrored copy of that array too. If you don't have high performance needs you can get by with slower hard drives. You also have the 25gb,40gb, or 100gb top of rack switch. Well you want two of those for redundancy. It starts to quickly add up. Annual system maintenance could be lower but is still a cost. Number 1 reason I'm in a datacenter is to swap a dead drive.

Oct 2020, StorONE announced a 1PB All-Flash array for $499k.

Other examples which are close to their pricing. https://www.reddit.com/r/storage/comments/vdr6ql/pricing_exa...

Re: How Canva saves Amazon S3 costs

#135

Post by AntonCTO » Mon, Jun 19, 2023, 7:03 PM UTC

andrewstuart wrote: post #20

onion2k wrote: post #6

Earlier quoted context omitted.

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops This is the traditional “sell” for cloud computing and I don’t buy it at all. The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems. The clouds sales pitch is the if you go cloud then you…

This is precisely my experience and what I observe in companies that fall into the cloud trap. In addition, it is important not to overlook the significant transformation and upskilling process required, as well as the time needed to accomplish it.

Re: How Canva saves Amazon S3 costs

#136

Post by jabart » Mon, Jun 19, 2023, 7:05 PM UTC

hinkley wrote: post #120

jabart wrote: post #103

Earlier quoted context omitted.

Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.

The last place I worked that had that kind of capacity, I had access to the lobby of that building and that was about it. The server room I could access had space for maybe 15-20 racks and in fact had maybe 7, which is probably why it was always cold as fuck in there and I didn’t like being in there much. Then there was a contract which had a server room twice the size and mostly contained an AS/3 and a couple racks…

AWS has a talk about things like this. They were, no idea if they still do, setup an AZ based on MWH figure for most efficiency because any higher the power company doesn't like it. Our local DC the water cooling and refrigerant cooling is larger than the 5 generators they have. I've crammed a lot of servers in my rack and I'm out of power plugs but still under 2kwh of what we are on contract for. Dense servers have really messed with datacenters.

Re: How Canva saves Amazon S3 costs

#137

Post by bluedino » Mon, Jun 19, 2023, 7:07 PM UTC

pjc50 wrote: post #35

rat9988 wrote: post #4

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

Have a look at the Backblaze blogs, where they detail how they built a storage provider on the cheap.

and be prepared to write your own software stack

Re: How Canva saves Amazon S3 costs

#138

Post by AntonCTO » Mon, Jun 19, 2023, 7:12 PM UTC

motoxpro wrote: post #42

A lot of people in this thread should start consultancies that transition companies off the cloud. Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.

I do. Interested? :D On a small project, I managed to reduce costs by 70 times simply by transitioning and eliminating unnecessary features. This happened four years ago, and the project is still running smoothly. However, what I truly enjoy is providing an objective view of the actual cloud transition process to companies before they make any misguided decisions.

Re: How Canva saves Amazon S3 costs

#139

Post by activiation » Mon, Jun 19, 2023, 7:32 PM UTC

jaza wrote: post #48

I'm really surprised how much anti cloud opinion is being voiced here (but maybe I should have known better, after all these years on HN). Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.

If you want trouble and exorbitant fees, the cloud is for you...

Re: How Canva saves Amazon S3 costs

#140

Post by charcircuit » Mon, Jun 19, 2023, 7:34 PM UTC

Quarrel wrote: post #99

charcircuit wrote: post #93

Earlier quoted context omitted.

Have you even looked at S3 bandwidth pricing? I don't get how anyone using S3 for storing media or binaries can justify that outrageous price unless every download is connected to something making you money.

Well, step 1, have petabytes of data in S3, then don't pay the sticker price. Step 2, write articles like this as part of your newly contracted "engagement" with Amazon to help justify the lower price.

>step 1, have petabytes of data in S3, then don't pay the sticker price.

So pay Amazon a bunch for the privilege to have lower prices. How about just starting off somewhere where you don't have to commit to paying a lot of money to get a reasonable price?