I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…
didn't they EOL omnipath last year?
Ask HN: Is your company sticking to on-premise servers? Why?
121–130 of 782 posts
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#122I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms ). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita... . All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. F…
I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#123Earlier quoted context omitted.
Infrastructure. Datacenter costs are a relatively high percentage of revenue. No public facing services - only large clients.
It sounds like you are operating a cloud.....
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#124The company I work for actually develops and hosts an AWS clone for the Linux Foundation, but with very specific requirements. They have special needs that requires baremetal machines and "real" networking between them across 6+ NICs per server.
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#125I use a $5/mo DigitalOcean VPS droplet instead of AWS or other "cloud" service. I only have to host an analytics dashboard ( https://usertrack.net/ ), I don't need scaling and this way I know exactly how much I will pay. The resources are more than enough for my needs and I don't think it could be much cheaper even on the most optimized pay-per-minute of use cloud platforms. I also have some other APIs hosted in the…
Whenever I need to host something small and I’m trying to decide between DO and AWS I always ask myself. Would I rather be surprised by the bill or my website crashing from too much traffic? I almost always pick DO because I don’t want to mess something up and lose a few hundred dollars.
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#126We have around 20 servers in a colo center down the street. At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS ha…
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#127Earlier quoted context omitted.
> Hundreds of terabytes a day is really not that much How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters. > Also surprised about latency We've spent a surprising amoun…
> How much would it cost to move this across boxes in EC2? Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region. > One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it). You're not going to have a successful…
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#128Founder of a growing startup: Started with a cluster of Raspberry Pis and expanded onto an old desktop. Primarily did this for cost (raspberry pis alone were more powerful than a GCP $35/mo instance). Everything was fine until I needed GPUs/handling more traffic than those Raspberrys could handle. So I expanded by including cloud instances in my Docker Swarm cluster (tidbit: Using Traefik and WireGuard) So half on-pr…
At $35/month though GCP would only have to save you a half an hour of maintenance for it to be worth it though.
Re: Ask HN: Is your company sticking to on-premise servers? Why?
#129University research group here. Simply, _cost_ Our compute servers crunch numbers and data at > 80% util. Our servers are optimized for the work we have. They run 24/7 picking jobs from queue. Cloud burst is often irrelevant here. They deal with Terabytes or even Petabytes of moving data. I’d cry paying for bandwidth costs if charged €/GB. Sysadmin(yours truly) would be needed even if it were to be run in the cloud.…
1. Their use case is kinda different. The servers mostly run heavy CS research related stuff. E.g. they might have heavy CPU load and heavy traffic between they servers but they have less often heavy traffic to the "normal internet" (if they have heavy traffic to the outside it's normally to other research institutes which not seldom have dedicated wire connections).
2. They might run target specific optimized CPU or GPU heavy compute tasks going on for weeks at a time. This is really expansive in the cloud which is mostly focused in thinks like web services.
3. When they don't run such tasks in the research groups they want to allow their juniors to run their research tasks "for free". Which wouldn't work with a payment model as done in the cloud.
4. They don't want to relay on some external company.
Also I'm not sure are there even (affordable) cloud systems with compatible spec? (like with 4+TB of RAM, I'm not kidding this is a requirement for some kind of tasks or they will take way to long and requires additional complexity by using special data structures which support partial offline data in the right way, which can be very costly in dev time)??