41–50 of 181 posts
Real question, how to scale a business to petabytes of data without cloud? How is it done in practice?
Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.
Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…
Many aws services are hit or miss but s3 is just a marvel. Running geo-distributed 100s PB Ceph cluster with tiered storage isn't exactly trivial.
Seems crazy to run such large infrastructure on a major cloud. Tens of millions wasted. Canva is likely trapped in S3 never to exit. The cost of getting their data out makes it impossible. S3…. the Hotel California of the cloud. You can check in any time you like but you can never leave. S3 is 9 cents per GB egress fees. Cloudflare R2 charges zero egress fees. At list price of 9 cents per GB it would cost Canva about…
Earlier quoted context omitted.
10 Petabytes is 10,000 TB. That's only 454 spinning disk drives (22 TB each). Approximate this as 600 with Raid 5. You can easily fit this in 100 desktops. There's tons of OSS to manage all layers of this stuff now.
Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…
Earlier quoted context omitted.
usually you're right but the cost mentioned here is specific to BLOB data storage ... a very specific problem that can be isolated and run on external services and save the company lots of money.
^^^ The cloud vs. on-prem argument often seems to ignore the (enormous) middle-ground. Just because one portion of your architecture would do well to be run outside the cloud, doesn't mean you take out all the other parts you don't want to deal with yourself. Furthmore, "on-prem" might mean in your building, in someone else's building co-located and you rent space and control it, or in someone else's building where t…
- Consistency. Instead of saying "Oh look in AWS for this app, Azure for this one, and hetzner for this app, except it's test env is in AWS", it all just lives in AWS. It massively simplifies docs, onboarding, and reduces the amount of one-person specialised knowledge.
- Engineering Costs. Similar to above, but in terms of engineering, there's less to know and understand. Instead of needing to know how the AWS load balancer routes/connects to a VM somewhere else, and how that VM gets it's blob-storage-data from azure, we only need to understand AWS concepts.
- Vendor Lock In. Yeah, it's there. If we have a service that uses data from S3, there's egress costs from S3 to , but not with EC2. We've consciously accepted this lock in for the time being.
Now, we're a 50 person company so YMMV, but the above tradeoffs plus an "opinionated" setup in AWS (everything on ECS, logging to Cloudwatch, RDS for DB) drastically reduced the "ops" overhead on our side after the initial setup. If I started over, I'd make the same decisions again.
Earlier quoted context omitted.
At Canva's scale you'll spend hundreds of thousands a year, just hiring one of the engineers working on your home-grown solution. Multiple this by 50 or more just for the staff, before talking about hardware, cost of delivery speed, cost of bugs, etc.
Instead you can spend millions on paying top market dollar to AWS experts trying to work out your giant pile of spaghetti cloud infrastructure. Desperately trying to work out if they can delete this resource or that because who the hell knows which bit of critical code is using it. Or spending days trying to manage a chewing gum ball of IAM policies.
The point is that you will spend millions anyway. Might as well have something better than whatever home-grown dumpster fire your team will come up with.
You also seem very ignorant of how AWS works. I've never had to wonder if I could delete a resource or not, because all our resources are created via CloudFormation. Same for IAM policies.
Even if you don't use any other cloud tech in your stack, my advice is, use S3 (or equivalent)! That level of durability, availability, and scalability, is not trivial to pull off, and is best made somebody else's problem. Even when you're as big as Canva.
Earlier quoted context omitted.
Slight tangent from the point, however: friends don't let friends use RAID5. Putting this out as a PSA reminder as it's not often discussed any more given the prevalance of cloud storage. RAID5 gives you really the worst of all worlds. You get poor performance (n/4) due to the number of operations required and you get the fragility of a RAID storm that comes from a cascade RAID failure when a drive dies. A "stripe of…
What's your opinion on RAID6? Reasonable middle ground between cost and redundancy or added complexity with the same performance problems as RAID5?
A lot of people in this thread should start consultancies that transition companies off the cloud. Apparently, it would be easy for them to save tens of millions, in turn making millions for the consultant.
One only says this if they don't have a good understanding of the differences between capex & opex, the amount of money spent on tech labor & compliance projects, as well as the level of assurances AWS give you regarding durability of data. All of this in 2023 when we have almost 20 years of data showing more and more business moving to the cloud and staying in the cloud.
I think anti-cloud sentiment aligns with hacker mentality on decentralization=good (which it is), anti big corporation feelings (boo amazon) and so it leads people in these threads to make emotional arguments over something that is clearly going in the other direction. It's an excellent example of confirmation bias, trying to look for any and all justification to say the cloud is worse, when the majority of businesses have decided otherwise.