Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

221–230 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#221
post #203

Earlier quoted context omitted.

Wholeheartedly agree. I think AWS is moving in the right direction with Lightsail[0], which is a service very similar to DO droplets and includes transfer. Nice if you want to use AWS for like one or two other services, but I tend to still go with DO for small things. [0]: https://aws.amazon.com/lightsail/

That sounds interesting. By "moving in the right direction" do you mean that it's still in beta or not released yet? Or that it's just the first step of many to come?

Lightsail works well now. It's a little over three years old. In the first year or so after it was released, they were notorious for being slow in most regards compared to their peers (it launched using rebranded instances from AWS, and used spinning disks, going up against SSDs their competitors were all using). They've largely caught up on performance with DigitalOcean, Linode, Vultr and similar.

That said, I've stuck with DigitalOcean even though Lightsail tests fine. I've had a great experience over the years with DO and see no reason to leave.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#222
post #50

We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc. We don't generally trust cloud providers to meet our requirements for: * uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots…

Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands. Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is…

Computation over Petabytes of data sounds fairly expensive, jobs that I was running over close to a PB could cost hundreds of dollars. Am I misremembering, doing it wrong, or underestimating your teams’s cloud budget?

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#223

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…

> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out"…

Umm, most workloads aren't so compute intense that cpu interconnect speed would have a huge impact. The vast majority of rented computer tasks will be for things that need to communicate with external services (payments, region synchronizing, general api state comms, etc). In these cases, I doubt that your code is tuned tightly enough to reveal interconnect latency (as opposed to cache misses and swaps). Besides, hooking up interconnect runs gets ludicrously complicated and expensive fast. Like, $50k in cables and a week of experimenting with numa zones and kernel settings for a small cluster. I can only imagine the headaches of orchestration on national lab clusters.

Source: I have set up, and currently manage a small cluster which is always pegged. I also collaborate with folks at a couple of the national labs for scientific research.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#225
I work in gamedev. Build servers, version control, etc. are almost always on-premise, even if a lot of other stuff has been transitioned to the cloud. There's a few reasons:

1) Bandwidth. I routinely saturate my plebian developer gigabit NIC links for half an hour, an hour, longer - and the servers slurp down even worse. In an AAA studio I am but one of hundreds of such workers. Getting a general purpouse internet connection that handles that kind of bandwidth to your heavily customized office is often just not really possible. If you're lucky your office is at least in the same metro area as a relevant datacenter. If you're really lucky you can maybe build a custom fiber or microwave link without prohibative cost. But with those kinds of geographical limitations, you're not so much relying on the general internet, so much as you're expanding your LAN to include a specific datacenter / zone of "the cloud" at that point.

2) Security. These servers are often completely disconnected from the internet, on a completely separate network, to help isolate them and reducing data exfiltration when some idiot installs malware-laden warez, despite clear corporate policy threatening to fire you if you so much as even think about installing bootleg software. Exceptions - where the servers do have internet access - are often recent, regrettable, and being reconsidered - because of, or perhaps despite, draconian whitelisting policies and other attempts at implementing defense in depth.

3) Customizability. Gamedev means devkits with strict NDAs and physical security requirements, and a motley assortment of phone hardware, that you want accessible to your build servers for automatic unit/integration testing. Oddball OS/driver/hardware may also be useful for such testing. Sure, if you can track down the right parties, you might be able to have your lawyers convince their lawyers to let you move said hardware into a datacenter, expand the IP whitelists, etc... but at that point all you've really done is made it harder to borrow a specific popular-but-discontinued phone model from the build farm for local debugging when it's the only one reproducing a specific crash when you want to debug and lack proper remote debug tooling.

...there are some inroads on the phone farms (AWS Device Farm, Xamarin Test Cloud) but I'm unaware of farms varied desktop hardware or devkits. Maybe they exist and just need better marketing?

I have some surplus "old" server hardware from one such gamedev job. Multiple 8gbit links on all of them. The "new" replacement hardware often still noticably bottlenecked for many operations.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#226

Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…

[offtopic] Awesome seeing you here! I used to be super active on Hypixel when I was in high school (I was actually #1 on the leaderboards for one of the games for over a year). One of my first large scale programming projects ever was creating a Hypixel mod for my guild. Years later I am now a software engineer working at Google on the YouTube algorithm!

Great to hear! Lots of the people who learned how to code because of Minecraft have ended up all over the industry. It's where I got my start, as well as many of my coworkers. That's part of our passion for making Hytale, to empower the next generation of youth who want a game they can tinker with and use to learn. If you haven't seen it yet definitely go take a look, got lots of cool plans for moddability and customization :)

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#227

Earlier quoted context omitted.

There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe. Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multi…

I'm curious why you moved to LLNL from HFT?

Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#228
I work for a large video games publisher, as you might expect we use a lot of windows.

Windows server licenses on AWS and GCP are hundreds of times more expensive at our scale. Incidentally we actually do have some cloud infra and we like it, but the licensing cost is half the total price of the instance itself.

In fact, you might not know this but games are relatively low margin, and we have accidentally risked the companies financial safety by moving into the cloud.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#229

Earlier quoted context omitted.

This is kind of a big caveat (“Netflix is in the cloud but almost none of the work is done there”), and something I have to mention to non tech decision makers when they say “but Netflix!”. I even have a slide for presentations just for this (“You Are Not Netflix”).

99% of the work is done on the cloud. What comes off of those colo servers is literally just bits streaming from disk to network. There is no transformation or anything. No authentication, no user accounts, no database. Nothing. Just static files served efficiently.

For models where static files need to be served efficiently, using cloud compute and aggressive CDNs works great. For any model where you've got volatile data that can't be cached (Gaming, VOIP, etc) you're tough out of luck.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#230

Earlier quoted context omitted.

In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.

This is accurate in my experience. Also keeping things up and being there in case shit hits the fan is a full-time job. You can't write features and also manage servers equally well. Unless you have no life I guess.

Totally. I've had personal colo'd servers for 20 years at this point. But I'm tired of knowing that at any point I might have to wake up and haul my ass down to San Jose to swear at some piece of failing gear. I'm excitedly moving it all into the cloud.
Post reply on HN