511–520 of 782 posts
Earlier quoted context omitted.
Cost. We currently pay less than €5000 monthly for 500TB/month in traffic and 50 Ryzen CPUs. Amazon would be $30,000 traffic + $100,000 compute.
I think you forgot to include cost of engineers in this calculation.
There is no way you need that many for maintaining a fleet of 50 CPUs (chance is that in OPs case they got duel socket servers as well).
Earlier quoted context omitted.
@nemothekid Can you provide a source for your claim on Uber's real estate costs for their data centers? I couldn't find anything on Google that corroborates your assertion.
Uber 10-K has operating lease payments in 2019 of $196M and in 2020 of $216M. That's not all data centers but a lot of it is data centers.
Earlier quoted context omitted.
Snap was spending 2B over 5 years - so 400M/yr. By comparison, Uber, who operates under managed colo, spends nearly 200M/yr alone on real estate for their datacenters. Who knows how much they are paying in engineering salaries so manage those datacenters. Personally I don't think there's a one size fits all solution, you will have to do the math (like I'm sure Snap, Netflix and others have done) to see if cloud is wo…
Unless that's capex for building, or includes server capex that they depreciate, that seems way high given Uber's scale. A pretty dense cabinet should only cost ~$1400/mo at wholesale (1MW+ rooms) rates, and $200M is 143,000 cabinets. And it's public that Uber uses multiple clouds as well. Disclaimer: I haven't reviewed public Uber filings, would be very interested if there's any data that indicates they're really sp…
Earlier quoted context omitted.
IMO, the cloud providers' main advantage is that they will professionally manage all of the underlying hardware. Optionally, they can also manage the low level pieces of software also (e.g. databases, file store, etc.). It's a safe bet that GCP, AWS, Azure et al. can manage/architect their data center much better than the vast majority of companies.
The last point is something that “we”, as an industry, can often be blind to. If your business is something that is fundamentally not “pushing bytes”, you really, really don’t want to know about routers, firewalls, the OSI layer, SHA256, RAID arrays, all the way to this week’s JS framework. All of that is a big annoyance, and paying AWS to “take care of it” makes sense, even if it comes at a higher price: more often…
Imagine a company in Europe that decides to host it's files on Alibaba Cloud in the US.
Imagine the US Department of State hosting it's files with Google.
Imagine an energy company working on new reactor Tech, ...
Imagine a Certificate Authority which has an offline store of root certificates which need to come online to sync twice a day.
Imagine cases where you need a hardware HSM.
Then there is also Cost as others have pointed out. AWS cost structure is so complex that whole business models[1] have sprung up to help you optimize the price tags or reduce the risks of huge bills. that's right: you need to have a commercial agreement with another partner that has nothing to do with your cloud just to work around aggressive pricing. The guy who started this ~2 years ago has grown to 40+ people (organically), is based in Switzerland and is still hiring even in this recession. It should give you an idea of how broken the cloud is.
Lastly there is also the lock-in. All the hours that you have to sit down and learn how the AWS IAM works is wasted once you decide to move to another cloud. The cost for learning how to use the 3rd party API is incurred by you not the cloud vendor. For people who think lock-in isn't much of a problem remember your whole hiring strategy will be aligned to whatever cloud vendor you're using (look at job description that already filter out based on AWS or GCP experience). Lock-in is so bad that for a business it is close to the rule of real-estate (location, location, location), only that it's to the advantage of the cloud vendor not you as the customer.
[1] optimyze.cloud
[2] "I have just tried to pull the official EC2 price list via JSON API, and the JSON file is 1.3gb" https://twitter.com/halvarflake/status/1258161778770542594
Earlier quoted context omitted.
Can you share the architecture and stack? What form do these jobs have? How do you manage workloads? How do you manage resources? Do users have quota for compute and storage? Do you use GPUs? If so, how do you deal with malfunction? Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster? What if a job requires dependencies? Do you create a compute environment on the fly or…
I wrote about it here: https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Y…
Earlier quoted context omitted.
> Those engineers' salaries almost always outweigh all of your cloud costs I think there is a divide here. Most people in this thread that mention cost as an issue are running some serious gear, and that does not come cheap. Your parent is running 70 servers plus some serious networking equipment, which is easily a couple million dollars. And he said that a cloud provider would 10x that cost. If all you need is a lit…
That reads as fairly dismissive to me. Most people running serious gear have serious engineering payroll overhead. Sure, some folks are managing something fairly static like game servers, but many of us work for bigco with two dozen products and massive teams. Even with thousands of ec2 instances we continue to move our on-prem infrastructure to the cloud because developer productivity saves us big money in the long…
I’ve heard stories of companies having each PR spin up something like 20 ec2 instances to build up a whole deployment. A CICD design like that used to be a fireable offense. Now people see that and assume it’s saving them money because it would have bottlenecked before.
Redundant power, redundant internet connections, and a few racks of Dell servers and gigabit switches. Why did I mention Dell? They just don't seem to die. We used HP for a few years but had a few bad experiences.
Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…
I remember a post here a while back from a guy running Bitcoin mining in his dorm room. One day he realized he could offer his spare cycles to grad students with high computing workloads and undercut cloud computing prices while increasing what he made far over just mining. It's very weird that we haven't fully arbitraged $/instruction to a single (low) price yet (or storage/hosting, whichever). If only there were an…