451–460 of 782 posts
Though we have adopted something close to an "Edge Computing" solution... I guess it comes down to "Why not both?" :)
I think it also depends on your definition of "server"
Yes. Why? Cost, availability, flexibility, bandwidth. For a lot of companies, on-prem servers are the best solution for efficiency and cost. One great example. We were paying $45k/yr for a hosted MS Dynamics GP solution. For $26k we brought it in house with only a $4k/yr maintenance fee. We bought a rackmount Dell, put on VMWare, have an app VM and a DB VM. My team can handle basic maintenance. In the past 11 months…
> The Cloud is just someone else's computer. If they're running those machines more cheaply than you are, they're cutting out some cost. The question is, do you need what they're cutting? They're cutting overhead and getting better deals on hardware than you could ever get. Their efficiency is their profit margin.
Last time I checked, AWS had a profit margin in the 40%-50% ballpark.
Sorry, but the semiconductor industry doesn't operate with any kind of markup that would allow such profit margins from "getting better deals on hardware". The only one able to make that kind of profit used to be Intel on high-end server CPUs, and even they are now pressured by AMD and custom ARM silicon options. Anything else needed for a server, RAM or flash chips or whatever, is usually selling on thin single-digit margins.
Cloud provider profit margin is perfectly logical and explainable through lock-in effects keeping their customers paying big markups to stay in AWS infrastructure. Be it software that was built against AWS proprietary services, be it having the necessary engineering skills to manage AWS infrastructure in the team but lacking the skills to manage on-prem hardware, be it the enterprise sales teams of cloud operators schmoozing CTOs of big corporations and making them jump on a "going into cloud" strategy as some kind of magic bullet to future-proof their corporations' IT, be it the psychological effect that makes "using the cloud" apparently a mandatory thing to be "cool" in todays' silicon valley culture, and therefore by extension the whole worlds' IT engineering culture.
The most ironical of them all is this weird effect that drives people to rationalize these things, writing comments like yours, because nobody likes to admit they've painted themselves into a corner of lock-in effects. And of course there's the irony of this all being history repeating itself: anyone still remembering when IBM dominated the IT industry?
We're concerned about corporate espionage and infiltration, so we can't trust our servers being out of our sight. Most people don't have the code on their physical machines either; I'm a cocky breath of fresh air in that regard in that I prefer my stuff to run locally instead of the (slow, underpowered) VMs, I trust Apple's encryption a lot.
Earlier quoted context omitted.
Same here - CTO of medium sized company. Our IT infra costs are 1/10th the cost of cloud, simply because I happen to be comfortable having on-premise machine and working on them (sometime myself). We have two dozen servers in two locations. It's more time to setup, but maintenance is actually quite low.
24 servers, interesting. That seems like a small enough setup that I would definitely have left it in the cloud. Much more than that and i think it makes sense to start moving to physical servers. But 24 I would have guessed would be cheaper to maintain in EC2. Are you setup with one or two racks in each location and maybe one full time IT/Sys admin at each location?
Earlier quoted context omitted.
The big question is - how much do you trust these providers.
I would guess that approximately zero banks own data centers that are operated by their own employees. There might be a few exceptions to this, but the reality is that most banks don't view technology as part of their core business. So this largely gets outsourced to IT consulting firms like Infosys, IBM, Wipro, etc. The big question is - how much do you trust these providers, and do you think they are more competent…
Earlier quoted context omitted.
Yes, you’re paying for a ton of stuff there that you probably don’t use and then are susceptible to bugs that have nothing to do with your use case. At $5B it would not cost anywhere near that much to replicate. Your infra would be better tailored to your workloads and your sw teams which would further drive costs down. The upsides are that you only drive features you need and keep things simple. The downside is that…
Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters. Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is wo…
Also Uber seems to be much less computing/BW demanding than Snap
Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…
> about 700 rented dedicated machines to service 70k-100k concurrent players Slightly off-topic, but it sounds like a single machine can't handle more than 150 concurrent players. How is this so? Is Hypixel Minecraft that resource intensive? What's the bottleneck — it is the CPU, or RAM, or the network latency(?) per machine, or is it something else? > We push about 4PB/mo in egress bandwidth With 70k players, 4PB /…
You're pretty accurate with the per player bandwidth. We measure it at an average of 200kbps/online player. We are incredibly compute-intensive, though, and these machines are all operating with E3-1271v3 CPUs, 32GB of RAM, and ~100GB SSD.
Earlier quoted context omitted.
A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and…
You’re not paying Google $5B for raw infra, you’re paying for cloud services like top-tier horizontally scalable databases, global availability, CDNs, datastores of different flavors, and transparently managed monitoring and hardware fault resolution.
One secondary factor is that we've only monotonically increased, and it's way cheaper to keep 10%-15% overprovisioned than to be on burst price with 50%+ constant load.
But the simplest math is - we have > 100 storage servers that are 2u 26x2tb flash, 256gb RAM, 36 cores. They cost $18k once, which we finance at pretty low interest over 36 months (and really last longer than that). Factor in $200-400/mo to host each depending (I think it's more like $200, but it doesn't matter for the cloud math).
That same server would be many $thousands/month on any cloud we've seen. Probably $4-6k/mo, depending on the type of EBS-ish attached. Or with the dedicated server 'alternate' they are moving to offer (and Oracle sorta launched with).
It'd be cheaper but still > 2x as expensive on Packet, IBM dedicated, OVH, Hetzner, Leaseweb (OVH and Hetzner the cheapest probably).
Three other factors for us:
1) Bandwidth would be outrageous on cloud but probably not as outrageously high as just the servers, given that our outbound is just our SaaS portal/API usage
2) We'd still need a cabinet with router/switch infra to peer with a couple dozen customers that run networks (other SaaS/digital natives and SPs that want to send infrastructure telemetry via direct network interconnect).
3) We've had 5-6 ops folks for 3 of the 6 years, 3-4 for the couple years before that. As we go forward, as we double we'll probably +1. It is my belief that we'd need more people in ops, or at least eng+ops mix, if we used public cloud. But in any case, the amount of time we spend adding and debugging our infra is really really really small, and the benefit of knowing how servers and switching stuff fails is huge to debugging (or, not having to debug).
All that said - we do run private SaaS clusters, and 100% of them are on bare metal, even though we could run on cloud. Once we do the TCO, no one yet has wanted to go cloud for an always-on data-intensive footprint like ours.
Good luck with your journey, whichever way you go!
And happy to discuss more, email in profile