Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

Post reply on HN
Print view View original

441–450 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#441

Post by danmaz74 »

tjbiddle wrote: post #241
horsawlarway wrote: post #217

I'm slowly coming to the complete opposite opinion you seem to have. I've worked almost entirely for companies that run services in various cloud infrastructures - Azure/Heroku/Aws/GCP/Other. I recently started a tiny 1 man dev shop in my spare time. Given my experience with cloud services it seemed like a no brainer to throw something up in the cloud and run with it. Except after a few months I realized I'm in an in…

What's the time trade-off? I've been drawing out my plans lately for a hobby project, all 100% on AWS. Being able to spin up my entire infrastructure with Terraform, build out images with Packer, setup rules for off-site backups, ensure everything is secure to the level I want it, etc. - It takes me next to no time at all. I can't imagine buying hardware, ensuring my home is setup with proper Internet, configuring ev…

Consider bare metal at eg hetzner.de - they do everything that is hardware related, but still the cost is way lower than aws.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#442

Post by boredpudding »

llarsson wrote: post #349
RachelF wrote: post #340

Earlier quoted context omitted.

Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage. My guess is they did it for cost reasons.

That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.

Since the move to S3 is only 'a few years ago', my guess is this was not the reason for them to move away from S3 since service at Dropbox is pretty much the same.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#443

Post by 2rsf »

dathinab wrote: post #155
kondro wrote: post #84

Earlier quoted context omitted.

But there are literally 100's (or more) of banks around the world that use AWS for compute. They meet all the security requirements you could want (27001, SOC2, and many others) around how they secure and limit access to servers by staff and have a variety of WORM audit tools and a very comprehensive IAM to help you satisfy these requirements are being met.

I don't know about the parent comments work place, but I don't think any non-US bank should use a AWS (or GCP or Azur). It's way to risky that changes in local and/or US regulations will require you to move your whole infrastructure to a new provider like in a month or two (which is completely impossible). Btw. same applies for any bank relying on the infrastructure of a foreign company. Like e.g. just think about th…

I'm European and was surprised to hear that legal issues are indeed a part of the decision, not just technical details.

AFAIK data that passes through the US may be legally intercepted by the US authorities.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#444

Post by adev_ »

hprotagonist wrote: post #53

AI compute is so much cheaper on-prem that it's not even in question. And there are clients that demand it. And researchers, in general, like to do totally wacky things, and it's often easier/cheaper to let us if you have physical access.

+1 that's one of the reason for us too.

GPU machines are clearly too expensive in most cloud providers compared to on-prem.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#445

Post by winter_blue »

dahfizz wrote: post #274
RhodesianHunter wrote: post #237

Earlier quoted context omitted.

> Cloud is great if your workload is variable and erratic Or if you're just constantly iterating on a large product with many engineers. Those engineers' salaries almost always outweigh all of your cloud costs and so making them productive is cost effective. Things like SNS/SQS/S3/VPC/ELB/etc. save you countless hours and often make up for the increased cloud costs with increased developer productivity.

> Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an issue are running some serious gear, and that does not come cheap. Your parent is running 70 servers plus some serious networking equipment, which is easily a couple million dollars. And he said that a cloud provider would 10x that cost. If all you need is a lit…

If one could come up with an estimate of how many hours per year it would cost to maintain (plus initially setup cost), and multiple that by $75/hr (a reasonable rate for a good engineer), then you could estimate how much it all costs.

I'd wager a guess that a properly-configured server won't actually cost that much to maintain. Unless you're frequently updating the OS and other stuff - but that's a software issue. I don't expect the hardware will fail that frequently at all. If there's a HDD failure, a good RAID system (with tolerance for 3 or 4 disks failing simultaneously) will alert you to it, and you'd grab the spare HDD (you should have a few spares), and pop it in, and the RAID would recover from the failure. What other hardware components frequently fail? RAM? CPU? Not really.

The engineer who sets up the server should be someone who is a generalist, and a full-time employee who when not tending to the server, works on other stuff (like the product). Then you have someone on staff who is familiar with the system, and can fix something that goes wrong, without needing to ramp up on it first. I know a lot of programmers would love to set up a server. Any who enjoys buildings PCs (which many generalist programmers do) would probably love to have a one-month project where they pick parts and set up a powerful server.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#446

Post by winter_blue »

AgentK20 wrote: post #207

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

> about 700 rented dedicated machines to service 70k-100k concurrent players

Slightly off-topic, but it sounds like a single machine can't handle more than 150 concurrent players. How is this so?

Is Hypixel Minecraft that resource intensive?

What's the bottleneck — it is the CPU, or RAM, or the network latency(?) per machine, or is it something else?

> We push about 4PB/mo in egress bandwidth

With 70k players, 4PB / 70k is approximately ~57 GB per player per month. That divided by the number of seconds in a month (2.628e+6) is: 57 x 10^9 / 2.628e+6 = ~21.7 kB.

Over a month, on average, a single player consumes a bandwidth of ~21.7 kB/sec, or about 174 kbps. This is an extraordinarily low per-player bandwidth consumption. At 150 players, each machine would average 26 Mbps of network traffic. This is fairly low as well. Of course, the machines have to be capable of handling of possibly much higher peak usage, but even an order of magnitude more — 260 Mbps, is something a Raspberry Pi 4 (which support full-throughput Gigabit Ethernet) can do.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#447

Post by mconstantinescu »

Hi @agentK20 , I agree with you that cloud providers should not take the skin out of you. I work for Sardina Systems, an ISV, and it is very frustrating for me to hear your high costs feedback more and more often everywhere. Please drop me an email at mconstantinescu@sardinasystems.com about your cloud details and I will make sure to provide you the best cost for it, very competitive in the market, as we are very flexible to adjust to any workload size and personalise the offer accordingly. Thank you, Mihaela

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#448

Post by irrational »

It is so so so much cheaper. We moved to AWS and tried setting up our servers with specs that were comparable to what our physical servers had been. We just about died after the first month. Our bill was higher for one month than for multiple years running our physical servers. We had to way dial them back to get the run rate down to a reasonable number. Now we have serious buyers remorse. Performance is terrible. The cost is still more per month than we ever had with our physical servers by a large amount.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#449

Post by avifreedman »

nemothekid wrote: post #305
mmmBacon wrote: post #292

Earlier quoted context omitted.

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that…

Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters. Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is wo…

Unless that's capex for building, or includes server capex that they depreciate, that seems way high given Uber's scale.

A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets.

And it's public that Uber uses multiple clouds as well.

Disclaimer: I haven't reviewed public Uber filings, would be very interested if there's any data that indicates they're really spending $200M on opex for real estate (which would be equivalent to $400M/year on cloud, which is either opex or potentially mix if there is some reserved instance-type cloud spend).

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#450

Post by soniman »

kevindong wrote: post #310
mmmBacon wrote: post #292

Earlier quoted context omitted.

Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that…

@nemothekid Can you provide a source for your claim on Uber's real estate costs for their data centers? I couldn't find anything on Google that corroborates your assertion.

Uber 10-K has operating lease payments in 2019 of $196M and in 2020 of $216M. That's not all data centers but a lot of it is data centers.
Post reply on HN