761–770 of 782 posts
The cost benefits are huge and since our app is mostly a normal web app we dont need that many fancy cloud things. And i dont see us needing it in the future.
I really dont understand why a company doing similar things would wanna go the could route. Its so damn expensive and its not always easy to use and setup.
Earlier quoted context omitted.
How much do you think a competent System engineer should be paid?
I think that's way too hard to answer since it depends on so many factors. But again, my experience isn't that the pay is too low. It's not like there are 10 people coming to the interview with no one taking the offer because of the salary. It's more like no one have applied for the job in three months and the external recruiters sniping LinkedIn didn't find anyone. But to give some kind of estimate just so you know…
Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…
Earlier quoted context omitted.
Which dedicated infra host did you use?
We bounced around through a few different hosts in the first two or three years of our operation, but for the past four years we've been consistently with SingleHop, now an INAP company. They've been great through the entire time that we've worked with them and have saved our asses on more than one occasion, especially back before mitigation tech like CloudFlare Spectrum was available.
( Interesting there is no official words on the website. But that is what Wiki shows. )
Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwid…
A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn’t scale cost-effectively on AWS. I figured out that we could colocate and…
Earlier quoted context omitted.
Can you share the architecture and stack? What form do these jobs have? How do you manage workloads? How do you manage resources? Do users have quota for compute and storage? Do you use GPUs? If so, how do you deal with malfunction? Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster? What if a job requires dependencies? Do you create a compute environment on the fly or…
I wrote about it here: https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Y…
Now, we thought of several ways of solving this. We have a branch that uses Kubernetes but this is not the one that is deployed.
We know about SLURM and this is something we will support for a coarse granularity. i.e: notebook level jobs. This is a common scenario and we'll need it for our AppBook (we allow users to turn a notebook into an application with one click and we automatically generate the form fields for the features, and the API endpoint for the model).
However, I'm interested in finer granularity computation; as in: I want a job for the cell. This involves looking into Jupyter kernels to circumvent the front-endkernel disconnection (we'll have to do that for another use case in low internet bandwidth scenario).
>Sorry, I could not understand this question. Do you mean metadata about A job?
How do your users upload, and manage data access and is the data mutable and versioned. What version of the code ran on which version of the data, and who changed that data, etc.
How do users access the data from a Jupyter notebook? Is the data in different object storages? Do you proxy the requests and handle access, things like that.
Earlier quoted context omitted.
I wrote about it here: https://aravindh.net/post/sysadmin/ > What form do these jobs have? Mostly batch jobs written as bash scripts. Occasionally, some users run singularity containers. But, all through SLURM. > How do you manage workloads? SLURM > How do you manage resources? As a sysadmin, my inventory is via Ansible. All activity on servers happen via Ansible only. > Do users have quota for compute and storage? Y…
Yeah, cloud dev-ops and HPC are so similar and still oh-so different (I run a smaller HPC cluster for genomics). When reading these questions, someone with an HPC background would probably already know the answers (or just accept SLURM as an answer for everything). With respect to the original questions -- I've investigated running a full SLURM cluster on the cloud and it just never seems worth the effort. For larger…
Lots of great insights here, which fully accord with my experience, even in the small end of town. About a year ago, I was in a meeting with my new CEO (who had acquired my company). My side of the business had kept hardware in-house, his was in AWS. We had broadly similar businesses in the same industry and with the same kind of customers. My side of the business needed to upgrade our 5+ year old hardware. The quote…
The big pitch is opex vs capex. This is one of the leading "value propositions" being made to the C-suite. If you're a business that's worried about capex then opex model will look very appealling... until someone looks at how much you'll (generally) be screwed in the Opex model in the mid to long term. Most 'decision makers' in large corporations are short term oriented and the opex model ends up winning.
Earlier quoted context omitted.
[offtopic] Awesome seeing you here! I used to be super active on Hypixel when I was in high school (I was actually #1 on the leaderboards for one of the games for over a year). One of my first large scale programming projects ever was creating a Hypixel mod for my guild. Years later I am now a software engineer working at Google on the YouTube algorithm!
Great to hear! Lots of the people who learned how to code because of Minecraft have ended up all over the industry. It's where I got my start, as well as many of my coworkers. That's part of our passion for making Hytale, to empower the next generation of youth who want a game they can tinker with and use to learn. If you haven't seen it yet definitely go take a look, got lots of cool plans for moddability and custom…
The downside is that this competes with school (thanks god he is very good so that's fine-ish)
The upside is that it is quarantine in France so I can manage this.
Thanks for all of that!
Earlier quoted context omitted.
> Firstly, if you are big enough, you can manage/architect your data centre better than a cloud provider. You can afford to hire staff to do it, and they can build something specific to your needs. Companies like this should be on-prem. It's interesting that competent engineers is only a question about cost to you. Where I am at (north of Europe) it's really hard to find good engineers. Even though it doesn't mean mu…
> The pay well and have great benefits but it's just not enough people out there to take these jobs and manage a full on-prem infrastructure. I’m from a small province in Canada and we have similar problems here sometimes. One of the questions I sometimes have to ask clients when they make a statement like that is: “do you pay well for the area ? Or do you pay well enough to attract talent from out-of-province?” This…
p.s hope you're staying safe from COVID