Live data from Hacker News

Move Back to Datacenters

rakkhi.substack.com

41–50 of 97 posts

Re: Move Back to Datacenters

#41

Move stuff back on-prem. Great. But aside from all the usual practical matters (power, backups, spare parts, service contracts, etc...), software is rapidly becoming a roadblock to move back to on-prem. If everything you have is running on Kubernetes or other open source software, great. If not, then there's an increasing amount of roadblocks being put up: * Some software is only available as SaaS anymore * Jacking u…

> Some software is only available as SaaS anymore

This reminds me of the idea of "Service as a Software Substitute" (SaaSS): https://www.gnu.org/philosophy/who-does-that-server-really-s...

> What does it mean to say that a given computing activity is your own? It means that no one else is inherently involved in it. To clarify the meaning of “inherently involved,” we present a thought experiment. Suppose that any software tasks you might need for the job are implemented in free software and you have copies, and you have whatever data you might need, as well as computers of whatever speed, functionality and capacity might be required. Could you do this particular computing activity entirely within those computers, not communicating with anyone else's computers?

> If you could, then the activity is entirely your own. For your freedom's sake, you deserve to control it. If you do it by running free software, you do control it. However, doing it via someone else's service would give that someone else control over your computing activity. We call that scenario SaaSS, and we say it is unjust.

It sounds a bit preachy like some of the FSF/GNU stuff sometimes does, but at the same time I can't shake the feeling that they're somewhat right and that those arguments have also fallen on deaf ears most of the time.

For my own personal stuff, I just self-host everything I need in containers and mostly use free software, so I can (and have) easily move hosting providers or host things myself in my homelab, have full backups and so on. But at the same time, it definitely takes a bit of work to figure everything out and not everyone wants that, especially in environments where you can point a finger and go "the vendor is having some issues, we're off the hook".

Plus, I don't think there are many alternatives to certain software out there, like MongoDB (maybe JSON in RDBMS if you really need to, but the API would be different then), or S3 (MinIO has a license that's too strong for some, Zenko feels abandoned, Garage is the same as MinIO, maybe SeaweedFS but that doesn't have full/mostly full API compatibility from looking into it).

Honestly, it sometimes unironically feels like building boring software that relies on MariaDB/PostgreSQL and maybe Redis/RabbitMQ is the right answer. Not quite sure about blob storage, though.

Re: Move Back to Datacenters

#42
With IAM you can restrict all dynamodb endpoint access to a VPCE/private link. But the insecurity of the public facing endpoints is vastly overstated even without using VPCE.

The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired.

I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance.

I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.

Re: Move Back to Datacenters

#43

So many negative comments. Makes you wonder who works for aws…

Everyone here talks up the cloud. They’ve all drunk the kool aid and are cloud true believers.

Also, a lot of people here work for startups or have their own startups.

The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.

Re: Move Back to Datacenters

#44
post #8

The analogy of this article would be: Stop using airbnbs when traveling, just build a house.

I think people overestimate the expenses related to maintaining their own infrastructure. I work for a large org that does both. We have a lot of workloads running in the public cloud, and many more on premises (for various reasons, including compliance). Having worked on these for years, building solutions, diagnosing problems and so on I'd risk saying we have reached the point where the cloud is slowly getting more…

This. A dozen time this.

I am lost in trying to start off a sensible / official / approved AWS setup but I can use some years old bash and salt s riots to setup my raspberry pis sitting there in front of me.

It's not just me (maybe mostly me) but there is a new layer of complexity - I can learn and understand the FOSS but trying to learn AWS means piercing the layers of marketing that are creeping in - one of the huge advantages of a FOSS readme is it comes from the head of the person who designed the system i want to use. not a product manager who probably misses the technical point anyway

So yeah. I am wary.

Re: Move Back to Datacenters

#45
post #28

Earlier quoted context omitted.

> Intentionally knee capping on-prem software feature wise Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's no…

Would you mind giving an example of something that is easy in cloud but hard on prem (all I can think of is scaling up (down?) which seems kinda obviously hard)

Scaling is definitely one, but another big category is just depending on other cloud services, whether it's message queues, managed databases, object storage, ML training pipelines, software-defined networking, whatever.

In any of those cases, we either have a choice to implement and support some onprem alternative, require the customer to provide the necessary service locally (which is not particularly effective for all sorts of reasons), or simply disable features onprem.

In the cloud, we get significant benefits from using managed services. Onprem, we not only lose those benefits, we have to pay an additional cost in terms of making our code support both environments and building out the deployment and support of the services in question.

It's really exactly the same issues that make cloud attractive to businesses in the first place.

Re: Move Back to Datacenters

#46

Move stuff back on-prem. Great. But aside from all the usual practical matters (power, backups, spare parts, service contracts, etc...), software is rapidly becoming a roadblock to move back to on-prem. If everything you have is running on Kubernetes or other open source software, great. If not, then there's an increasing amount of roadblocks being put up: * Some software is only available as SaaS anymore * Jacking u…

> Some software is only available as SaaS anymore This reminds me of the idea of "Service as a Software Substitute" (SaaSS): https://www.gnu.org/philosophy/who-does-that-server-really-s... > What does it mean to say that a given computing activity is your own? It means that no one else is inherently involved in it. To clarify the meaning of “inherently involved,” we present a thought experiment. Suppose that any soft…

It's not just services like databases or other APIs applications hook into, it's the applications themselves, like accounting software or Microsoft Exchange and Office 365, etc...

Re: Move Back to Datacenters

#47
post #27

The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure, but also having to manage a complex platform stack and its security. The security points, which the main points hinges on, seem hyperfocused and in most cases misguided. Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere. The CPU attack example given is for AM…

The other thing they keep missing. Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired. There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents? It is easy to say that people can mov…

[deleted]

Re: Move Back to Datacenters

#48

The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure, but also having to manage a complex platform stack and its security. The security points, which the main points hinges on, seem hyperfocused and in most cases misguided. Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere. The CPU attack example given is for AM…

Genuinely feels like there’s a constant trickle of arguments from people looking to do all they can to shift the tides such that their skill set is more in demand. Or maybe it’s just an ideological obsession. I don’t know. I feel the same way whenever the “use C for everything” people catch a whiff of performance mattering and very strongly suggest that the only way to dig ourselves out of this mess is to move back t…

I believe both arguments held some merits. It all depend on the scale.

If you are a startup and/or have a limited/specialized business scope* Cloud can be the best choice.

For organizations of 4000+ people that manage 100+ applications (from different vendors) with a 90+ people IT department this is maybe not the silver bullet. At this scale you need to keep complexity and security manageable and a more traditional redondant datacenter architecture might be less of a headache than the cloud.

*PS: What I mean by "limited/specialized business scope" is for instance Twitter/Meta. Huge scale but heavily specialized around their main products and develop most of theirs application in house.

Re: Move Back to Datacenters

#49
No. We rolled our own stack for 10 years, until 2018, well after AWS and Azure were around.

We switched to Azure in 2018 and never looked back.

Sure you trade security (do you really, though?) in exchange for:

- not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one

- not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us

- not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes

and the list goes on ad infinitum.

This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.

Re: Move Back to Datacenters

#50
post #8

The analogy of this article would be: Stop using airbnbs when traveling, just build a house.

No, the analogy would be: stop staying in the same airbnb for a year, just build a house.

Or:

You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.

Post reply on HN