Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

21–30 of 181 posts

Re: How Canva saves Amazon S3 costs

#21

Post by jokethrowaway »

Canva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s

More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud.

At Canva's scale you'll spend hundreds of thousands, save literal millions and have way more headroom and stability than with crappy auctioned instances on AWS.

Re: How Canva saves Amazon S3 costs

#22

Post by aljarry »

acchow wrote: post #5
rat9988 wrote: post #4

Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

But that's on a single RAID in a single location. You'd probably need at least one failover, and backups for that as well, if you'd like to target higher availability / durability.

S3 keeps multiple copies in multiple geographic locations.

Re: How Canva saves Amazon S3 costs

#23

Post by buildbot »

simonjgreen wrote: post #13
acchow wrote: post #5

Earlier quoted context omitted.

10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…

Or “software” raid 10 with ZFS mirror dev’s all slapped into one pool?

Re: How Canva saves Amazon S3 costs

#24

Post by iLoveOncall »

jokethrowaway wrote: post #21

Canva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud. At Canva's scale you'll spend hundred…

At Canva's scale you'll spend hundreds of thousands a year, just hiring one of the engineers working on your home-grown solution. Multiple this by 50 or more just for the staff, before talking about hardware, cost of delivery speed, cost of bugs, etc.

Re: How Canva saves Amazon S3 costs

#25

Post by onion2k »

andrewstuart wrote: post #20
onion2k wrote: post #6

Earlier quoted context omitted.

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops This is the traditional “sell” for cloud computing and I don’t buy it at all. The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems. The clouds sales pitch is the if you go cloud then you…

Don’t believe it.

You don't need to 'believe'. You need to do an analysis of what you need and how much each option will cost. Then you can know.

If your argument is "I believe onprem saves money" or "I bet AWS is cheaper" or "Jim Morrison came to me in a dream and said I should use Azure" then you haven't done enough research.

Re: How Canva saves Amazon S3 costs

#26

Post by andrewstuart »

iLoveOncall wrote: post #24
jokethrowaway wrote: post #21

Canva is a case where AWS made sense for the founders. The founders anyway made billions and who cares if you give a few hundreds millions to Jeff and have to hire expensive AWS consultants if you can avoid hiring a few ops people that actually know how to operate a server /s More seriously, I think the sweet spot is to buy 2x o 3x what you expect to need but do it off the cloud. At Canva's scale you'll spend hundred…

At Canva's scale you'll spend hundreds of thousands a year, just hiring one of the engineers working on your home-grown solution. Multiple this by 50 or more just for the staff, before talking about hardware, cost of delivery speed, cost of bugs, etc.

Instead you can spend millions on paying top market dollar to AWS experts trying to work out your giant pile of spaghetti cloud infrastructure.

Desperately trying to work out if they can delete this resource or that because who the hell knows which bit of critical code is using it. Or spending days trying to manage a chewing gum ball of IAM policies.

Re: How Canva saves Amazon S3 costs

#27

Post by ramraj07 »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Do you know how much it’ll cost to store 230 petabytes per month on premises at the same redundancy and availability level? I have done such calculations for academia at a smaller scale and unless you literally want it to last decades it’s definitely not worth it to do your own infra on this.

Re: How Canva saves Amazon S3 costs

#28

Post by xarope »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

let me give some hypothetical; let's say Canva decides to cut cost by cutting the amount of storage users could have (e.g. quota, or lifetime):

- if they had a sunk cost CAPEX of 100 PB of storage, all they could do is sell the storage servers for a fraction of the original cost (and folks like me snapping up pre-owned server equipment for said fraction of cost for our home labs).

- on the other, since S3 is OPEX (not sure if they are locked in to 1 year billing cycles? but still better than 3-5 year CAPEX runs), they could cut the cost of that storage "immediately".

Similarly, if they needed to expand, CAPEX means placing an order for the storage, waiting for delivery, racking, burning in etc, versus just paying almost "instantly" for more storage.

So long as they are still earning money, why wouldn't they stick to this OPEX model?

Not saying this is good or bad, it depends on your business model, ultimately.

Re: How Canva saves Amazon S3 costs

#29

Post by orangepanda »

Offtopic — is it some cardinal sin to include a link to the home page from a documentation? I dont know what canva does, if I stumble on a documentation page first, should I not be allowed to see the product's landing page at some point? This is so prevalent, it has to be a conscious choice.

Re: How Canva saves Amazon S3 costs

#30

Post by ToJans »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

I'd go for a hybrid approach: new data is stored on private cloud, and old data would be gradually migrated or phased out.
Post reply on HN