Back in the ancient era of the mainframes, this "multitenancy" concept would have been called "time sharing". It looks like everything old is new again.
If you stay around long enough the pendulum swings full circle.
How the economics of multitenancy work
31–40 of 49 posts
Re: How the economics of multitenancy work
#32In my experience scaling dynamically just makes things slower and it doesn't reduce costs significantly compared to having dedicated resources. Resourcing dynamically is also difficult because you don't actually know upfront how many resources your CI needs.
It can when workloads are relatively predictable day-over-day but have low lows and high peaks. For example,
My team has a service which has a daily traffic peak that is >5x our traffic min.
Scaling on traffic and resource demand gives us an increased average utilization rate for the hardware we pay for. Especially when peaks are short lived, an hour out of 24 for example.
Re: How the economics of multitenancy work
#33Back in the ancient era of the mainframes, this "multitenancy" concept would have been called "time sharing". It looks like everything old is new again.
I was kind of disappointed the first time I saw an IBM mainframe and it kinda just looked like a rack of servers. To be fair, it was taking up a little bit of a server room that had clearly been designed for a larger predecessor and now almost had enough free space for a proper game of ping pong. Hyperscaler rack designs definitely blur this line further. In some ways I think Oxide is trying to reinvent the mainframe…
Nah, oxide is standardizing pluggable, scalable, rack-unit API driven local cloud, with extremely tight integration that nobody else has largely except Apple.
There's different takes on it, that's just mine. I really appreciate and respect their work.
Re: How the economics of multitenancy work
#34Am I reading this right, they are going to rack their own servers for this business?
If I were them I would be looking at renting from bargin bin hosting providers like hetzner or ovh to run this on. The great thing is that hetzner also has a large pool of racked servers that you can tap into.
You are basically going to re-implement hetzner at a smaller (and probably worse) scale by creating your own multitenant mini cloud for running these ci jobs.
Free advice: set up a giant kubernetes cluster on hetzner/ovh, use gvisor runtime for isolation, submit ci workloads as k8s jobs, give the k8s jobs different priority classes based on job urgency and/or some sort of credit system, jobs will naturally be executed/preempted based upon priority.
There you go, that is the product using nearly 100% existing and open source software.
Re: How the economics of multitenancy work
#35Back in the ancient era of the mainframes, this "multitenancy" concept would have been called "time sharing". It looks like everything old is new again.
I was kind of disappointed the first time I saw an IBM mainframe and it kinda just looked like a rack of servers. To be fair, it was taking up a little bit of a server room that had clearly been designed for a larger predecessor and now almost had enough free space for a proper game of ping pong. Hyperscaler rack designs definitely blur this line further. In some ways I think Oxide is trying to reinvent the mainframe…
My school district growing up had a decently sized Unisys mainframe. While I was working there, they upgraded to a new machine.
The old machine was about desk height and 15 feet wide. The new machine was a 4u box running a Unisys mainframe emulator on NT 4, on a quad processor pentium pro. Pretty sad. But they did keep the giant line printer, at least while I was working there.
Re: How the economics of multitenancy work
#36Back in the ancient era of the mainframes, this "multitenancy" concept would have been called "time sharing". It looks like everything old is new again.
Fascinating how the wording has changed from sharing to tenancy. Maybe reading a bit more into it, but isn't it funny how the modern word evokes the landlord/tenant relationship?
(I'm reading into this further than it needs to go for fun, primarily)
Re: How the economics of multitenancy work
#37Great write-up. OP's next steps are probably to offer off-peak capacity at a discount. If you pass along some of the savings to your customers, they'll happily take their once-daily or once-weekly jobs and make sure they get scheduled off-peak instead of getting scheduled at some time that was either random or arbitrary. Win-win.
Re: How the economics of multitenancy work
#38> We have a fleet of hundreds of bare-metal gaming CPUs
Curious what you use and why. Larger datacenter CPUs have a steep entry price, but usually better economics. Also, don't trust the public pricing — it's totally broken in this industry, unfortunately.
Re: How the economics of multitenancy work
#39For the last two years, I've been running github actions like this:
- start a paused vm in google cloud
- run the build there (via a gcloud ssh command) and capture the output
- pause the vm after it is done
Takes about 4-5 minutes. Maybe a few dozen times per month. It's a nice fast machine with lots of CPU and memory. Would cost a small fortune to run 24x7. It would cost more than it costs to run our entire production environment. But a few hours of build time per month barely moves the needle.
Our build and tests max out those CPUs. We only pay for the minutes it is running. Without that it would takw 2-3 times as long. And it would sometimes fail because some of our async tests time out if they take too long.
It's not the most elegant thing I've ever done but it works and hasn't failed me in the two years I've been using this setup.
But it's also a bit artificial because bare metal is cheaper and it runs 24x7. The real underlying issue is the vastly inflated price of virtual machines cloud providers rent out vs. the cost of the hardware that powers them. The physical servers pay themselves back within weeks of coming online. Everything after is pure profit.
Re: How the economics of multitenancy work
#40For the last two years, I've been running github actions like this: - start a paused vm in google cloud - run the build there (via a gcloud ssh command) and capture the output - pause the vm after it is done Takes about 4-5 minutes. Maybe a few dozen times per month. It's a nice fast machine with lots of CPU and memory. Would cost a small fortune to run 24x7. It would cost more than it costs to run our entire product…
very interesting idea!
How much cheap is this as compared to github actions?
also why are you using gcloud? would certain other competitors like aws/(hetzner? if we are talking about vps) also suit the case.
I would love it if you could write a blog post about it.