Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

351–360 of 516 posts

Re: Don't rent the cloud, own instead

#351

Earlier quoted context omitted.

I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy s…

A tricky thing on this site is that there are lots of different people with very different kinds of experience, which often results in people talking past each other. A lot of people here have experience as zero-to-one early startup engineers, and yep, I share your experience that Heroku was very popular in that space. A lot of other people have experience at later growth and infrastructure focused startups, and they…

Absolutely true but I also think it’s a fair callout when the intent was to disprove the original post asking how old someone was because 15 years ago everyone was stringing together their own services which is absolutely not true. There were many shades of gray at that time both in my experience of either have a sysops/devops team to help or deploying to Heroku as well as folks that were indeed stringing together services.

I find it equally disingenuous to suggest that Heroku was only for startups with lavish budgets. Absolutely not true. That’s my only purpose here. Everyone has different experiences but don’t go and push your own narrative as the only one especially when it’s not true.

Re: Don't rent the cloud, own instead

#352

Earlier quoted context omitted.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on…

You are right. Are you actually seriously considering whether to go fully managed or self managed at this point? Pls go AWS route and thank me later :)

Re: Don't rent the cloud, own instead

#353
The cloud is a psyop, a scam. Except at the tiniest free-tier / near free-tier use cases, or true scale to zero setups.

I've helped a startup with 2.5M revenue reduce their cloud spend from close to 2M/yr to below 1M/yr. They could have reached 250k/yr renting bare-metal servers. Probably 100k/yr in colos by spending 250k once on hardware. They had the staff to do it but the CEO was too scared.

Cloud evangelism (is it advocacy now?) messed up the minds of swaths of software engineers. Suddenly costs didn't matter and scaling was the answer to poor designs. Sizing your resource requirements became a lost art, and getting into reaction mode became law.

Welcome to "move fast and get out of business", all enabled by cloud architecture blogs that recommend tight integration with vendor lock-in mechanisms.

Use the cloud to move fast, but stick to cloud-agnostic tooling so that it doesn't suck you in forever.

I've seen how much cloud vendors are willing to spend to get business. That's when you realize just how massive their margins are.

Re: Don't rent the cloud, own instead

#354

Earlier quoted context omitted.

I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy s…

> Heroku was wildly popular with startups The world's a lot bigger than startups

Did you fail to finish reading the rest? At the same time I had touch with organizations that were still in data centers but I as an engineer had no touch on the bare metal and ticket systems were in place to help provision necessary services. I was not deploying my own Postgres database.

Your original statement is factually incorrect.

Re: Don't rent the cloud, own instead

#355
post #316

Everything comes circle. Back in my day, we just called it a "data center". Or on-premise. You know, before the cloud even existed. A 1990s VP of IT would look at this post and say, what's new? Better computing for sure. Better virtualization and administration software, definitely. Cooling and power and racks? More of the same. The argument made 2 decades ago was that you shouldn't own the infrastructure (capital ex…

> you shouldn't own the infrastructure (capital expense) and instead just account for the cost as operational expense (opex) That was part of the reason. The real reason was the internal infrastructure team in many orgs got nowhere. There was a huge queue and many teams instead had to find infinite workarounds including standing up their own. The "cloud" provided a standardized way to at least deal with this mess e.g…

I agree - my reference to the 1990s VP of IT was looking at the post, which is about on-premise data centers... not the cloud. I don't think there's a speed advantage for on-premise data centers now vs the 1990s, but if there is let me know. Otherwise, indeed, it's a 1990s-era blast from the past.

Re: Don't rent the cloud, own instead

#356
post #224

Earlier quoted context omitted.

No amount of money will make me maintain my own dbs. We tried it at first and it was a nightmare.

It's worth becoming good at.

I really do not think so. Most startups should rather focus on their core competency and direct engineering resources to their edge. When you are $100 mln ARR then feel free to mess around with whatever db setup you want.

Re: Don't rent the cloud, own instead

#357
Ah Slurm, so good to see it still being used. As soon as I touched it in ~2010 I realized this was finally the solid queue management system we needed. Things like Sun Grid Engine or PBS were always such awful and burdensome PoS.

IIRC, Slurm came out of LLNL, and it finally made both usage and management of a cluster of nodes really easy and fun.

Compare Slurm to something like AWS Batch or Google Batch and just laugh at what the cloud has created...

Re: Don't rent the cloud, own instead

#358

The cloud is a psyop, a scam. Except at the tiniest free-tier / near free-tier use cases, or true scale to zero setups. I've helped a startup with 2.5M revenue reduce their cloud spend from close to 2M/yr to below 1M/yr. They could have reached 250k/yr renting bare-metal servers. Probably 100k/yr in colos by spending 250k once on hardware. They had the staff to do it but the CEO was too scared. Cloud evangelism (is i…

> The cloud is a psyop, a scam.

You're just young.

> Suddenly costs didn't matter and scaling was the answer to poor designs.

It did.

Did you know that cloud cost less than what the internal IT team at a company would charge you?

Let's say you worked on product A for a company and needed additional VM. Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the cloud.

> Sizing your resource requirements became a lost art

In what way? We used to size for 2-4x since getting additional resources (for the in-house team) would be weeks to months. Same old - just cloud edition.

Re: Don't rent the cloud, own instead

#359
post #206

Earlier quoted context omitted.

Heroku was popular with startups who didn’t have infrastructure skills but the price was high enough that anyone who wasn’t in that triangle of “lavish budget, small team, limited app diversity” wasn’t using it. Things like AWS IaaS were far more popular due to the lower cost and greater flexibility but even that was far from a majority service class.

I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy s…

I have no doubt that was your experience. My point was that it wasn’t even common in SV as whole, just the startup scene. Think about headcount: how many times fewer people worked at your startup than any one of Apple, Oracle, HP, Salesforce, Intuit, eBay, Yahoo, etc.? Then thing about how many other companies there are just in the Bay Area who have large IT investments even if they’re not tech companies.

Even at their peak, Heroku was a niche. If you’d gone conferences like WWDC or Pycon at the time, they’d be well represented, yes, and plenty of people liked them but it wasn’t a secret that they didn’t cover everyone’s needs or that pricing was off putting for many people, and that tended to go up the bigger the company you talked to because larger organizations have more complex needs and they use enough stuff that they already have teams of people with those skills.

Re: Don't rent the cloud, own instead

#360

Earlier quoted context omitted.

And if each of your clients is in the Healthcare industry and dealing with end-user medical data? Or financial data? Are you prepared for appropriate data isolation/sharding and controls? Do you have a strategy for scaling database operations per client or across all clients? It really depends on the business model as to how well you might support your own infrastructure vs. relying on a new backend instance per clie…

> And if each of your clients is in the Healthcare industry and dealing with end-user medical data? Or financial data? Then you're probably going to need some combination of HIPAA / SOC 2 / PCI DSS certification, regardless of where your servers are physically located. AWS has certified the infrastructure side for you, but that doesn't remove your obligations for the logical side. > Are you prepared for appropriate d…

  >> vs. relying on a new backend instance per client in a cloud infrastructure

  > You want to spin up an EC2 per client, and run an isolated copy of the application, isolated DB, etc. inside of it? That sounds like a nightmare to manage, especially if you want or need HA capabilities.
No... just running a new hosted database instance per client... but (re)using your service/application infrastructure, but just connecting through a different database host/proxy based on the client for the request.

Just that utility at the database management layer is probably worth the price of entry for using cloud resources if you can't justify and cover the cost of say 5+ employees just for the data management infrastructure.

Post reply on HN