Live data from Hacker News

Ask HN: Is your company sticking to on-premise servers? Why?

news.ycombinator.com

121–130 of 782 posts

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#121

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…

didn't they EOL omnipath last year?

Yes but we still have many OP systems on the floor (see the link above).

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#122

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…

I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.

thanks!

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#123
post #77

Earlier quoted context omitted.

Infrastructure. Datacenter costs are a relatively high percentage of revenue. No public facing services - only large clients.

It sounds like you are operating a cloud.....

Sure... but at least they're not using the cloud! Just a whole bunch of servers!

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#124
There are some things you just can't do in a VM.

The company I work for actually develops and hosts an AWS clone for the Linux Foundation, but with very specific requirements. They have special needs that requires baremetal machines and "real" networking between them across 6+ NICs per server.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#125
post #34

I use a $5/mo DigitalOcean VPS droplet instead of AWS or other "cloud" service. I only have to host an analytics dashboard ( https://usertrack.net/ ), I don't need scaling and this way I know exactly how much I will pay. The resources are more than enough for my needs and I don't think it could be much cheaper even on the most optimized pay-per-minute of use cloud platforms. I also have some other APIs hosted in the…

Whenever I need to host something small and I’m trying to decide between DO and AWS I always ask myself. Would I rather be surprised by the bill or my website crashing from too much traffic? I almost always pick DO because I don’t want to mess something up and lose a few hundred dollars.

Wholeheartedly agree. I think AWS is moving in the right direction with Lightsail[0], which is a service very similar to DO droplets and includes transfer. Nice if you want to use AWS for like one or two other services, but I tend to still go with DO for small things.

[0]: https://aws.amazon.com/lightsail/

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#126

We have around 20 servers in a colo center down the street. At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS ha…

I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can definitely be easier to manage your own hardware than to manage your cloud infrastructure.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#127

Earlier quoted context omitted.

> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amoun…

> How much would it cost to move this across boxes in EC2? Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). You're not going to have a successful…

You pay for cross-AZ Traffic in AWS, and that adds up really fast.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#128
post #38

Founder of a growing startup: Started with a cluster of Raspberry Pis and expanded onto an old desktop. Primarily did this for cost (raspberry pis alone were more powerful than a GCP $35/mo instance). Everything was fine until I needed GPUs/handling more traffic than those Raspberrys could handle. So I expanded by including cloud instances in my Docker Swarm cluster (tidbit: Using Traefik and WireGuard) So half on-pr…

At $35/month though GCP would only have to save you a half an hour of maintenance for it to be worth it though.

You write this as if it's unrealistic. Who spends half an hour or more every month managing each tiny server? The closest server to me right now has probably taken about that long in the past year, swapping out a failed drive that unfortunately wasn't in a hot-swap bay. And it's much more powerful than a $35/month cloud instance.

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#129

University research group here. Simply, _cost_ Our compute servers crunch numbers and data at > 80% util. Our servers are optimized for the work we have. They run 24/7 picking jobs from queue. Cloud burst is often irrelevant here. They deal with Terabytes or even Petabytes of moving data. I’d cry paying for bandwidth costs if charged €/GB. Sysadmin(yours truly) would be needed even if it were to be run in the cloud.…

In germany it's a pretty common think for universities to have some servers for themself.

1. Their use case is kinda different. The servers mostly run heavy CS research related stuff. E.g. they might have heavy CPU load and heavy traffic between they servers but they have less often heavy traffic to the "normal internet" (if they have heavy traffic to the outside it's normally to other research institutes which not seldom have dedicated wire connections).

2. They might run target specific optimized CPU or GPU heavy compute tasks going on for weeks at a time. This is really expansive in the cloud which is mostly focused in thinks like web services.

3. When they don't run such tasks in the research groups they want to allow their juniors to run their research tasks "for free". Which wouldn't work with a payment model as done in the cloud.

4. They don't want to relay on some external company.

Also I'm not sure are there even (affordable) cloud systems with compatible spec? (like with 4+TB of RAM, I'm not kidding this is a requirement for some kind of tasks or they will take way to long and requires additional complexity by using special data structures which support partial offline data in the right way, which can be very costly in dev time)??

Re: Ask HN: Is your company sticking to on-premise servers? Why?

#130
We are software development company. Most of our compute/storage needs are for build/test cycles. We recently bought ~100K worth of additional hardware to migrate some of that work off AWS. The storage / virtualization is done using Ceph & OpenNebula. Including colocation/electricity/networking costs, the investment will pay for itself in ~9 months. If I would include deployment costs & work to migrate the jobs off the AWS -- it will pay for itself in 11 months.
Post reply on HN