Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Canva saves Amazon S3 costs

canva.dev

Post reply on HN
Print view View original

151–160 of 181 posts

Re: How Canva saves Amazon S3 costs

#151

Post by gurchik »

electroly wrote: post #79
andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of…

Cloudflare R2 also has a tool in beta that provides an incremental migration from S3. As users request files using your R2 URLs, Cloudflare automatically migrates the files from S3 to R2 on request. Depending on your user's request patterns this may be a way to migrate everything without paying any additional cost. If you have a lot of files that are rarely accessed then they will be slow to migrate, but because they are rarely accessed your S3 costs could be significantly reduced by using different storage classes anyway. Eventually you will have a small enough number of infrequently accessed files remaining in S3 that the additional cost to migrate them all in one go can make financial sense.

Re: How Canva saves Amazon S3 costs

#152

Post by colechristensen »

icedchai wrote: post #109
colechristensen wrote: post #9

Earlier quoted context omitted.

You can have a few petabytes in a single rack. 50-ish drives in a single 4U server is done.

What about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.

Plenty of tools for that on premises. Are you actually asking?

Re: How Canva saves Amazon S3 costs

#153

Post by simonjgreen »

AntonCTO wrote: post #147
simonjgreen wrote: post #13

Earlier quoted context omitted.

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…

RAID10 for 22TB disks with only two drives at "1" level? I would expect no less than 3 drives at such capacity. Otherwise there is a huge chance to lose the second drive while 1. changing the disc (=time, could be even days) and 2. replication. What are your thoughts on implementing RAID 0 over four RAID 1 arrays, each consisting of three disks, resulting in a total of 12 drives?

Agreed, 22TB is a _long_ rebuild time. I'd either go 3 drives in the mirror in that scenario, or go for smaller spindles, depending on requirements. It's not always about the biggest spindles possible.

Re: How Canva saves Amazon S3 costs

#154

Post by vkaku »

andrewstuart wrote: post #2

Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…

Cloudflare could build a product that not only caches, but transfers S3 to R2 on usage.

- Since non transactional data is likely to be on GitHub or somewhere, it should be easy to re-hydrate that on R2.

- Once enough time passes, there could be a sunset on unused data, or a strategy planned to 1-time exit write-off for S3 with those assets not on R2.

Standard migration procedure.

Re: How Canva saves Amazon S3 costs

#155

Post by simonjgreen »

wongarsu wrote: post #45
simonjgreen wrote: post #13

Earlier quoted context omitted.

Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…

What's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?

RAID6 is passable for a low performance small scale bulk storage, but the performance hit is worse than RAID5 and the rebuild times are even longer.

Re: How Canva saves Amazon S3 costs

#156

Post by vkaku »

gurchik wrote: post #151
electroly wrote: post #79

Earlier quoted context omitted.

Tip for anyone looking to spend seven-figure or more sums on one-time egress: Direct Connect egress is $0.02/GB. Rent a rack at a Direct Connect facility and get as many 10G fiber Direct Connects as you need, with corresponding flat rate 10G Internet ports with HE/Cogent/whatever transit provider. If you're going to be spending millions on egress, you could just hire someone to set this up for you. With that kind of…

Cloudflare R2 also has a tool in beta that provides an incremental migration from S3. As users request files using your R2 URLs, Cloudflare automatically migrates the files from S3 to R2 on request. Depending on your user's request patterns this may be a way to migrate everything without paying any additional cost. If you have a lot of files that are rarely accessed then they will be slow to migrate, but because they…

Awesome. I just wrote about it above. Thanks for sharing!

Re: How Canva saves Amazon S3 costs

#157

Post by simonjgreen »

zokier wrote: post #49
wongarsu wrote: post #45

Earlier quoted context omitted.

What's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?

With Ceph, and I imagine other similar solutions, you can skip RAID altogether and manage redundancy at higher level over plain disks. RAID makes sense if you specifically want to have conventional filesystem on top of it, but for object storage its not necessary.

You're applying the same principles, just at the object level rather than the filesystem level. You still have to choose a redundancy pattern.

But I do agree with you, it does make the problem significantly easier/different.

ZFS and Ceph are marvelous creations.

Re: How Canva saves Amazon S3 costs

#158

Post by jasmer »

andrewstuart wrote: post #20
onion2k wrote: post #6

Earlier quoted context omitted.

There's always a cheaper way to do something, but it's important to remember that 'lower price' often doesn't mean 'lower cost'. In order to get the lower price you need to spend on whatever the alternative option isn't doing in order to give that price saving. For example, moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops. And then yo…

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops This is the traditional “sell” for cloud computing and I don’t buy it at all. The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems. The clouds sales pitch is the if you go cloud then you…

[dead]

Re: How Canva saves Amazon S3 costs

#159

Post by icedchai »

colechristensen wrote: post #152
icedchai wrote: post #109

Earlier quoted context omitted.

What about single points of failure? Replication? High availability? Disaster recovery? Backup / snapshots/ versioning? This is one of the benefits of the cloud.

Plenty of tools for that on premises. Are you actually asking?

No, I'm just mentioning these things to show that the decision is more complex than a "4U server" vs "S3." S3/AWS gives you a ton of stuff you'd have to implement yourself. Plus you need to hire more people to manage it.

Re: How Canva saves Amazon S3 costs

#160

Post by kazen44 »

oxfordmale wrote: post #121
andrewstuart wrote: post #20

Earlier quoted context omitted.

>> moving from AWS to onprem means you need to configure the infrastructure yourself; you save on AWS fees but you spend more on devops This is the traditional “sell” for cloud computing and I don’t buy it at all. The clouds would have you believe it doesn't make sense to run your own systems because you need too much specialist expertise to run your own systems. The clouds sales pitch is the if you go cloud then you…

I challenge any in-house team to achieve the same reliability as S3. To be fair most businesses probably don't need it, but it is worth taking into account.

the same reliability at what scale? this is a major difference. S3 has amazing reliability but is a incredible complex system with many moving parts, which incurs cost in terms of reliability. The simpler the system, the more reliable it usually is. Most bussiness do not operate at even remotely the same scale as amazon, and running a small scale cluster of S3 like functionality wouldn't be that hard.
Post reply on HN