Not nearly on the article's level, but I've been operating what I call a fog machine (itsy bitsy personal cloud) for about 15 years. It's just a bunch of local and off-site NAS boxes. It has kinda worked out great. Mostly Synology, but probably won't be when their scheduled retirement comes up. The networking is dead simple, the power use is distributed, and the size of it all is still a monster for me - back in the day, I had to use it for a very large audio project to keep backups of something like 750,000 albums and other audio recordings along with their metadata and assets.
Don't rent the cloud, own instead
391–400 of 516 posts
Re: Don't rent the cloud, own instead
#392And now go do that in another region. Bam, savings gone. /s
What I mean is that I'm assuming the math here works because the primary purpose of the hardware is training models. You don't need 6 or 7 nines for that is what I'm imagining. But when you have customers across geography that use your app hosted on those servers pretty much 24/7 then you can't afford much downtime.
Re: Don't rent the cloud, own instead
#393This also depends so much on your scaling needs. If you need 3 mid-sized ECS/EC2 instances, a load balancer, and a database with backups, renting those from AWS isn’t going to be significantly more expensive for a decent-sized company than hiring someone to manage a cluster for you and dealing with all the overhead of keeping it maintained and secure. If you’re at the scale of hundreds of instances, that math changes…
I also thinks it’s risk model too. Every time I see these kind of posts I think it misses the point there is a balance not only on cost like you describe but risk as well. You are paying to offload some of the risk from yourself.
> You are paying to offload some of the risk from yourself.
The opposite is also true: one is risking being banned by exascalers.
Re: Don't rent the cloud, own instead
#394Am I the only one that is simply scared of running your own cloud? What happens if your administrator credentials get leaked? At least with Azure I can phone microsoft and initiate a recovery. Because of backups and soft deletion policies quite a lot is possible. I guess you can build in these failsafe scenarios locally too? But what if a fire happens like in South Korea? Sure most companies run more immediate risks…
Do you have a computer at home? Are you scared of its credentials leaking? A server is just another computer with a good internet connection. You can equip your server with a mouse, keyboard and screen and then it doesn't even need credentials. The credential is your physical access to the mouse and keyboard.
I mean people are nowadays are really scared of using microwave oven too. What happens if I heat my coffee 1 min too long. Could be near death experience. Thats why I always drive down to Starbucks for coffee!
Re: Don't rent the cloud, own instead
#395The cloud is a psyop, a scam. Except at the tiniest free-tier / near free-tier use cases, or true scale to zero setups. I've helped a startup with 2.5M revenue reduce their cloud spend from close to 2M/yr to below 1M/yr. They could have reached 250k/yr renting bare-metal servers. Probably 100k/yr in colos by spending 250k once on hardware. They had the staff to do it but the CEO was too scared. Cloud evangelism (is i…
> The cloud is a psyop, a scam. You're just young. > Suddenly costs didn't matter and scaling was the answer to poor designs. It did. Did you know that cloud cost less than what the internal IT team at a company would charge you? Let's say you worked on product A for a company and needed additional VM. Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the c…
> You're just young.
And I feel great!
> Did you know that cloud cost less than what the internal IT team at a company would charge you?
Yes. Internal IT teams ran old-school are inefficient. And that's what the vendor tells you while they create shadow IT inside your company. Skip ITSM and ITIL... do it the SRE way.
Until the cloud economist (real role) comes in and finds a way to extract more rent out of their customer base (like GCP's upcoming doubling rates on CDN Interconnect). And until internal IT kills shadow IT and regains management of cloud deployments. Cybersecurity and stuff...
Back to square one. ITIL with cloud deployments. Some use cases will be way cheaper... but for your 100s of PBs of enterprise data, that's another story. And data gravity will kill many initiatives just based on bit movement costs.
> Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the cloud.
To some extent. One is hard dollars the other is funny money. But I thought paying for cloud with the company credit card was a 2016 thing. Now it's paid through your internal IT cost center, with internal IT markup.
I've seen petabytes of data move to the cloud and then we couldn't perform some queries on it anymore as that store wouldn't support it, and we'd need to spend 7 figures to move to another cloud database to query it. And that's hard dollars.
Yes, during early cloud days it was lean and aimed at startups. Now it's aimed at enterprise, and for some reason lots of startups still think it's optimized for them. It's not and it hasn't been for a long time.
Re: Don't rent the cloud, own instead
#396[dead]
Re: Don't rent the cloud, own instead
#397>San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48” intake fans and dual 48” exhaust fans to keep the air cool. To ensure low humidity ( Oh man, this is bad advice. Airborn humidity and contaminants will KILL your servers on a very short horizon in most places - even San Diego. I highly s…
Yes, it's easy to destroy the servers with a lot of dust and/or high humidity. But with filtering and ensuring humidity never exceeds 45% we've had pretty good results.
Re: Don't rent the cloud, own instead
#398Earlier quoted context omitted.
Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.
Managing the PostgreSQL databases is a medium to low complexity task as I see it. Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery. We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'…
> We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of.
this part is actually scariest, since there are like 10 different 3rd party solutions of unknown stability and maintanability.
Re: Don't rent the cloud, own instead
#399Earlier quoted context omitted.
The complexity is what gets you. One of AWS's favorite situations is 1) Senior engineer starts on AWS 2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't) 3) New engineer comes in and panics 4) Ends up using a "managed service" to relieve the panic 5) New engineer leaves 6) Second new engineer comes in and not only…
One of the biggest problems with the self-hosted situations I’ve seen is when the senior engineers who set it up leave and the next generation has to figure out how to run it all. In theory with perfect documentation they’d have a good head start to learn it, but there is always a lot of unwritten knowledge involved in managing an inherited setup. With AWS the knowledge is at least transferable and you can find peopl…
There are many great developers who are not also SREs. Building and operating/maintaining have their different mindsets.
Re: Don't rent the cloud, own instead
#400Earlier quoted context omitted.
> The cloud is a psyop, a scam. You're just young. > Suddenly costs didn't matter and scaling was the answer to poor designs. It did. Did you know that cloud cost less than what the internal IT team at a company would charge you? Let's say you worked on product A for a company and needed additional VM. Besides paperwork, the cost to you (for your cost center) would be more than using the company credit card for the c…
> You're just young. And I feel great! > Did you know that cloud cost less than what the internal IT team at a company would charge you? Yes. Internal IT teams ran old-school are inefficient. And that's what the vendor tells you while they create shadow IT inside your company. Skip ITSM and ITIL... do it the SRE way. Until the cloud economist (real role) comes in and finds a way to extract more rent out of their cust…
> Yes. Internal IT teams ran old-school are inefficient.
They aren't. It's politics. They want to protect and improve their own headcount and resources.
> One is hard dollars the other is funny money.
All the same to a team / department. It's not like people run it like their own wallet.
> finds a way to extract more rent out of their customer base
I find you just have a grudge against the cloud and hence too young. For every example you have the so-called "internal" IT team can and will do just the same. Go back to 90s, 00s - it was the same. The infra team wanted some fancy new storage arrays and charge everyone 2x for the new service etc.
> and for some reason lots of startups still think it's optimized for them. It's not and it hasn't been for a long time.
The problem isn't the cloud. Startups have always worked like this even 10-20 years ago. It's about wastage. They can raise and grow faster. So they think. The problem, if any is recently money isn't as cheap. Nothing new.