Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't rent the cloud, own instead

blog.comma.ai

Post reply on HN
Print view View original

401–410 of 516 posts

Re: Don't rent the cloud, own instead

#401

Post by organsnyder »

toomuchtodo wrote: post #269
organsnyder wrote: post #263

Earlier quoted context omitted.

I've worked with many "enterprise IT" sysadmins (in healthcare, specifically). Some are very proficient generalists, but most (in my experience) are fluent in only their specific platforms, no different than the typical AWS engineer.

Perhaps we need bootcamps for on prem stacks if we are concerned about a skills gap. This is no different imho from the trades skills shortage many developed countries face. The muscle must be flexed. Otherwise, you will be held captive by a provider "who does it all for you". "Today, we are going to calculate the power requirements for this rack, rack the equipment, wire power and network up, and learn how to use PX…

This might be my own ego talking (I see myself as a generalist), but IMHO what we need are people that are comfortable jumping into unfamiliar systems and learning on-the-fly, applying their existing knowledge to new domains (while recognizing the assumptions their existing knowledge is causing them to make). That seems much harder to teach, especially in a boot camp format.

Re: Don't rent the cloud, own instead

#402

Post by JackSlateur »

redrove wrote: post #293
IFC_LLC wrote: post #243

This is cool. Yet, there are levels of insanity and those depend on your inability to estimate things. When I'm launching a project it's easier for me to rent $250 worth of compute from AWS. When the project consumes $30k a month, it's easier for me to rent a colocation. My point is that a good engineer should know how to calculate all the ups and downs here to propose a sound plan to the management. That's the winni…

We’re past the point in history where most engineers get to make even a recommendation about which platform to use to management. In 99.999999% of cases management has already decided and is just informing you, because they know better.

I work in a multi-billions dollars company and do not face what you describe

Perhaps an exception (yet so far, I've never encounter the situation you describe)

Re: Don't rent the cloud, own instead

#403

Post by JackSlateur »

lelanthran wrote: post #52
jillesvangurp wrote: post #29

At scale (like comma.ai), it's probably cheaper. But until then it's a long term cost optimization with really high upfront capital expenditure and risk. Which means it doesn't make much sense for the majority of startup companies until they become late stage and their hosting cost actually becomes a big cost burden. There are in between solutions. Renting bare metal instead of renting virtual machines can be quite n…

Your calculation assumes that an FTE is needed to maintain a few beefy servers. Once they are up and running that employee is spending at most a few hours a month on them. Maybe even a few hours every six months. OTOH you are specifically ignoring that you'll require mostly the same time from a cloud trained person if you're all-in on AWS. I expect the marginal cost of one employee over the other is zero.

Maintainer is a real work

If you do it only a few hours every 6 months, you are not maintaining your infrastructure, you are letting it die (until the need arises and everything must be done and this is a massive project)

Re: Don't rent the cloud, own instead

#404

Post by kccqzy »

Torq_boi wrote: post #397
tgtweak wrote: post #385

>San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48” intake fans and dual 48” exhaust fans to keep the air cool. To ensure low humidity ( Oh man, this is bad advice. Airborn humidity and contaminants will KILL your servers on a very short horizon in most places - even San Diego. I highly s…

Yes, it's easy to destroy the servers with a lot of dust and/or high humidity. But with filtering and ensuring humidity never exceeds 45% we've had pretty good results.

I remember visiting a small data center (about half the size of the Comma one) where shoe covers were required. Apparently they were worried about people’s shoes bringing in dust and other contamination.

Re: Don't rent the cloud, own instead

#405

Post by sanderjd »

cheema33 wrote: post #381
Symbiote wrote: post #309

Earlier quoted context omitted.

Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…

> Managing the PostgreSQL databases is a medium to low complexity task as I see it. Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that. I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much…

For what it's worth, I have also managed my own databases, but that's exactly why I don't think it's a good use of my time. Because it does take time! And managed database options are abundant, inexpensive, and perform well. So I just don't really see the appeal of putting time into this.

Re: Don't rent the cloud, own instead

#406

Post by direwolf20 »

geodel wrote: post #394
direwolf20 wrote: post #144

Earlier quoted context omitted.

Do you have a computer at home? Are you scared of its credentials leaking? A server is just another computer with a good internet connection. You can equip your server with a mouse, keyboard and screen and then it doesn't even need credentials. The credential is your physical access to the mouse and keyboard.

I mean people are nowadays are really scared of using microwave oven too. What happens if I heat my coffee 1 min too long. Could be near death experience. Thats why I always drive down to Starbucks for coffee!

True! Decline of defiance or something. Everyone is suddenly a follower. Any idea what caused it? Micro plastics in the brain? Social media?

Re: Don't rent the cloud, own instead

#407

Post by sanderjd »

objektif wrote: post #352
sanderjd wrote: post #341

Earlier quoted context omitted.

This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on…

You are right. Are you actually seriously considering whether to go fully managed or self managed at this point? Pls go AWS route and thank me later :)

No not at all, I have the same opinion as you! But I'm curious to understand the opposite view.

Re: Don't rent the cloud, own instead

#408

Post by tgtweak »

Torq_boi wrote: post #397
tgtweak wrote: post #385

>San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48” intake fans and dual 48” exhaust fans to keep the air cool. To ensure low humidity ( Oh man, this is bad advice. Airborn humidity and contaminants will KILL your servers on a very short horizon in most places - even San Diego. I highly s…

Yes, it's easy to destroy the servers with a lot of dust and/or high humidity. But with filtering and ensuring humidity never exceeds 45% we've had pretty good results.

It's not a static number as it's also based on ambient air temperature in the form of dew point - 45% RH at low temps can be far more dangerous than 65% RH at warm ambient.

Likewise the impact on server longevity is not a finite boundary but rather "exposure over time" gradient that, if exceeding the "low risk" boundary (>-12'C/10'f dew point or >15'C/59'f dry bulb temp) results in lower MTBF than design. This is defined (and server equipment manufacturers conform and build to) ASHRAE TC 9.9. This mean - if you're running your servers above high risk curve for humidity and temperature, you're shortening the life considerably compared to low risk curve.

Generally, 15% RH is considered suboptimal and can be dangerous near freezing temperatures - in San Diego in January there were several 90%+RH scenarios that would have been dangerous for servers even when mixed down with warm exhaust air - furthermore, the outdoor air at 76'f during that period means you have limited capacity to mix in warm exhaust air (which btw came from that same 99%RH input air) without getting into higher-than-ideal intake temps.

Any dew points above 62.5'f are considered high risk for servers - as are any intake temps exceeding 32'C/90'f. You want to be on the midpoint between those and 16'C/65'f temps & -12'C/10'f dew point to have no impact on server longevity or MTBF rates.

As a recent example:

  KCASANDI6112 - January 2, 2026

                 High         Low      Average

  Temperature    73.4 °F   59.9 °F      63.5 °F

  Dew Point      68.0 °F   60.0 °F      62.6 °F

  Humidity       99 %      81 %         96 %

  Precipitation  0.12 in   --           --
Lastly, air contaminants - in the form of dust (that can be filtered out) and chemicals (which can't without extensive scrubbing) are probably the most detrimental to server equipment if not properly managed, and require very intentional and frequent filter changes (typically high MERV pleated filters changed on a time or pressure drop signal) to prevent server degradation and equipment risks.

The last consideration is fire suppression - permitted datacenters usually require compliance with separate fire code, such that direct outdoor air exchange without active shutdown and dry suppression is not permitted - this is to prevent a scenario where your equipment catches on fire and a constant supply of fresh oxygen-rich outdoor air turns that into an inferno. Smoke detection systems don't operate well with outdoor-mixed air or any level of airborn particulates.

So - for those reasons - among a few others - open air datacenters are not recommended unless you're doing them at google or meta scale, and in those scenarios you typically have much more extensive systems and purpose-designed hardware in order to operate for the design life of the equipment without issues.

Re: Don't rent the cloud, own instead

#409

Post by sanderjd »

infecto wrote: post #351
sanderjd wrote: post #277

Earlier quoted context omitted.

A tricky thing on this site is that there are lots of different people with very different kinds of experience, which often results in people talking past each other. A lot of people here have experience as zero-to-one early startup engineers, and yep, I share your experience that Heroku was very popular in that space. A lot of other people have experience at later growth and infrastructure focused startups, and they…

Absolutely true but I also think it’s a fair callout when the intent was to disprove the original post asking how old someone was because 15 years ago everyone was stringing together their own services which is absolutely not true. There were many shades of gray at that time both in my experience of either have a sysops/devops team to help or deploying to Heroku as well as folks that were indeed stringing together se…

I kind of thought the "15 years" was just one of those things where people kind of forget what year it is. Wow, 2010 was already over 15 years ago?? That kind of mistake. I think this person was thinking pre-2005. I graduated college just after that, and that's when all this cloud and managed services stuff was just starting to explode. I think it's true that before that, pretty much everyone was maintaining actual servers somewhere. (For instance, I helped out with the physical servers for our CS lab some when I was in college. Most of what we hosted on those would be easier to do on the cloud now, but that wasn't a thing then.)
Post reply on HN