Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use our own hardware

fastmail.com

Post reply on HN
Print view View original

251–260 of 547 posts

Re: Why we use our own hardware

#251

Post by irunmyownemail »

zbentley wrote: post #163
noprocrasted wrote: post #150

Earlier quoted context omitted.

3. "Dedicated server" at any hosting provider Open their management console, press order now, 15 mins later get your server's IP address.

For purposes of this discussion, isn't AWS just a very large hosting provider? I.e. most hosting providers give you the option for virtual or dedicated hardware. So does Amazon (metal instances). Like, "cloud" was always an ill-defined term, but in the case of "how do I provision full servers" I think there's no qualitative difference between Amazon and other hosting providers. Quantitative, sure.

Qualitatively, AWS is greedy and nickle and dime you to death. Their Route53 service doesn't even have all the standard DNS options I need and I can get everywhere else or even on my own running bind9. I do not use IPv6 for several reasons, when AWS decided charge for IPv4, I went looking elsewhere to get my VM's.

I can't even imagine how much the US Federal Government is charging American taxpayers to pay AWS for hosting there, it has to be astronomical.

Re: Why we use our own hardware

#252

Post by evantbyrne »

UltraSane wrote: post #235
evantbyrne wrote: post #223

Earlier quoted context omitted.

Making API calls from a VM on shared hardware to KMS is vastly more secure than doing AES locally? I'm skeptical to say the least.

Encrypting data is easy, securely managing keys is the hard part. KMS is the Key Management Service. And AWS put a lot of thought and work into it. https://docs.aws.amazon.com/kms/latest/cryptographic-details...

KMS access is granted by either environment variables or by authorizing the instance itself. Either way, if the instance is compromised, then so is access to KMS. So unless your threat model involves preventing the government from looking at your data through some theoretical sophisticated physical attack, then your primary concerns are likely the same as running a box in another physically secure location. So the same rules of needing to design your encryption scheme to minimize blowout from a complete hostile takeover still apply.

Re: Why we use our own hardware

#253

Post by motorest »

johnklos wrote: post #65

The whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding…

> All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding,(...)

This is where you lose all credibility.

I'm going to focus on a single aspect: performance. If you're serving a global user base and your business, like practically all online businesses, is greatly impacted by performance problems, the only solution to a physics problem is to deploy your application closer to your users.

With any cloud provider that's done with a few clicks and an invoice of a few hundred bucks a month. If you're running your hardware... What solution do you have to show for? Do you hope to create a corporate structure to rent a place to host your hardware manned by a dedicated team? What options f you have?

Re: Why we use our own hardware

#254

Post by sgarland »

noprocrasted wrote: post #173
briHass wrote: post #86

The biggest win with running your own infra is disk/IO speeds, as noted here and in DHH's series on leaving cloud ( https://world.hey.com/dhh/we-have-left-the-cloud-251760fb ) The cloud providers really kill you on IO for your VMs. Even if 'remote' SSDs are available with configurable ($$) IOPs/bandwidth limits, the size of your VM usually dictates a pitiful max IO/BW limit. In Azure, something like a 4-core 16GB RAM…

Biggest issue isn't even sequential speed but latency. In the cloud all persistent storage is networked and has significantly more latency than direct-attached disks. This is a physical (speed of light) limit, you can't pay your way out of it, or throw more CPU at it. This has a huge impact for certain workloads like relational databases.

Yep. This is why my 12-year old Dell R620s with Ceph on NVMe via Infiniband outperform the newest RDS and Aurora instances: the disk latency is measured in microseconds. Locally attached is of course even faster.

Re: Why we use our own hardware

#255

Post by dorongrinstein »

We at Control Plane (https://cpln.com) make it easy to repatriate from the cloud, yet leverage the union of all the services provided by AWS, GCP and Azure. Many of our customers moved from cloud A to cloud B, and often to their own colocation cage, and in one case their own home cluster. Check out https://repatriate.cloud

Re: Why we use our own hardware

#256

Post by qingcharles »

Beijinger wrote: post #207

I was told Fastmail is excellent, and I am not a big fan of gmail. Once locked out for good in gmail, your email and apps associated with it, are gone forever. Source? Personal experience. "A private inbox $60 for 12 months". I assume it is USD, not AU$ (AFAIK, Fastmail is based in Australia.) Still pricey. At https://www.infomaniak.com/ I can buy email service for an (in my case external) domain for 18 Euro a year a…

You can have as many domains as you want for free in your Fastmail account. There are no extra fees.

I've used them for 20 years now. Highly recommended.

Re: Why we use our own hardware

#257

Post by gausswho »

xiande04 wrote: post #236

Aside: Fastmail was the best email provider I ever used. The interface was intuitive and responsive, both on mobile and web. They have extensive documentation for everything. I was able to set up a custom domain and and a catch-all email address in a few minutes. Customer support is great, too. I emailed them about an issue and they responded within the hour (turns out it was my fault). I feel like it's a really matu…

I also chose Proton for the same reason. It hurts that their product development is glacial but that's a crucial component that I don't understand why Fastmail doesn't try to offer.

Re: Why we use our own hardware

#258

Post by graemep »

sgarland wrote: post #156
oftenwrong wrote: post #80

Earlier quoted context omitted.

In small companies, cloud also provides the ability to work around technical debt and to reduce risk. For example, I have seen several cases where poorly designed systems that unexpectedly used too much memory, and there was no time to fix it, so the company increased the memory on all instances with a few clicks. When you need to do this immediately to avoid a botched release that has already been called "successful…

> For example, I have seen several cases where poorly designed systems that unexpectedly used too much memory > using a managed database system like RDS vs self-managing the same RDBMS: using the managed version saves on labor As a DBRE / SRE, I can confidently assert that belief in the latter is often directly responsible for the former. AWS is quite clear in their shared responsibility model [0] that you are still…

Not a DB admin, but I do install and manage DBs for small clients.

My experience is that AWS makes the easy things easy and the difficult things difficult, and the knowledge is not transferable.

With a CLI or non-cloud management tools I can create, admin and upgrade a database (or anything else) exactly the same way, locally, on a local VM, and on a cloud VM from any provider (including AWS). Doing it with a managed database means learning how the provider does it - which takes longer and I personally find it more difficult (and stressful).

What I cannot do as well as a real DB admin could do is things like tuning. Its not really an issue for small clients (a few generic changes to scale settings to available resources is enough - and cheaper than paying someone to tune it). Come to think of it, I do not even know how to make those changes on AWS and just hope the defaults match the size of RDS you are paying for (and change when you scale up?).

having written the above I am now doubting whether I have done the right thing in the past.

Re: Why we use our own hardware

#259

Post by UltraSane »

wslh wrote: post #250
UltraSane wrote: post #151

Earlier quoted context omitted.

"All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding," This is false. AWS infrastructure is vastly more secure than almost all company data centers. AWS has a rule that the same person cannot have logical access and physical access to the same storage device. Very few companies have enough IT people to have this rule. The AWS KMS is vastly m…

From a critical perspective, your comment made me think about the risks posed by rogue IT personnel, especially at scale in the cloud. For example, Fastmail is a single point of failure as a DoS target, whereas attacking an entire datacenter can impact multiple clients simultaneously. It all comes down to understanding the attack vectors.

Cloud providers are very big targets but have enormous economic incentive to be secure and thus have very large teams of very competent security experts.

Re: Why we use our own hardware

#260

Post by sgarland »

noprocrasted wrote: post #165
TacticalCoder wrote: post #125

Earlier quoted context omitted.

> So for someone who may well be quite skilled at K8s administration ... And running your own hardware is not incompatible with Kubernetes: on the contrary. You can fully well have your infra spin up VMs and then do container orchestration if that's your thing. And part your hardware monitoring and reporting tool can work perfectly fine from containers. Bare metal -> Hypervisor -> VM -> container orchestration -> a c…

> And running your own hardware is not incompatible with Kubernetes: on the contrary Kubernetes actually makes so much more sense on bare-metal hardware. On the cloud, I think the value prop is dubious - your cloud provider is already giving you VMs, why would you need to subdivide them further and add yet another layer of orchestration? Not to mention that you're getting 2010s-era performance on those VMs, so subdiv…

> Not to mention that you're getting 2010s-era performance on those VMs, so subdividing them is terrible from a performance point of view too.

I was trying in vain to explain to our infra team a couple of weeks ago why giving my team a dedicated node of a newer instance family with DDR5 RAM would be beneficial for an application which is heavily constrained by RAM speed. People seem to assume that compute is homogenous.

Post reply on HN