> A “starter pack” 15amp cabinet with a gigabit connection can be rented for as little as $400 per month. Link?...
https://he.net/colocation.html
Can't speak about the power given on their $400/mo cabinet deal.
Rolling your own servers with Kubernetes
141–150 of 209 posts
Re: Rolling your own servers with Kubernetes
#142Earlier quoted context omitted.
How much would a 48U rack like described in the article cost a month? I would guess in the thousands if not ten thousands....
It mostly goes by power not rack units, if you have 48U and only 15amps, you're not going to fill it up. 15A is not a lot.
15A is more than you think these days, still probably need more for a full rack but a modern server can draw around 150W under full load these days. Hell, my entire rack at home hangs around ~.3A during normal usage (2x Dell R210 II's, Dell R320, Dell R520, Lenovo SA120 with 4 bays filled, Juniper EX2200-48T, Mikrotik CRS317).
Re: Rolling your own servers with Kubernetes
#143Earlier quoted context omitted.
I worked on Colossus at Google and Ceph is the closest thing out there. Gregory Farnum gave a great talk about it at the Open Source Summit 2017. I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.
I'm working on Rook+Ceph at Red Hat. Rook 1.0 was just released last week which added support for the very latest Nautilus release of Ceph.
Great to hear. I couldn't get more details from them, but it was something about both rebalancing and data recovery. They picked SwiftStack, which is at least in part open source. Maybe by now you have figured who that company is.
My team is still very interested in Ceph and Rook, for the record.
Re: Rolling your own servers with Kubernetes
#144The problem is not running kubernetes on "bare-metal". That's the somewhat easy and cheap part.
The problem is the supply chain to spec, purchase, rack, validate, provision and keep up to date your servers, their drivers and firmware. It's your capacity planning. It's keeping up with the rate of hardware evolution. What happens when you have to go from 10G to 25G networking ?
Oh and you also need a solid scalable L4 loadbalancing solution (facebook has a nice opensource dataplane based on ebpf)
You can build / assemble from open-source bits most of the classic IaaS bits (compute/storage/networkLB). It's doable, the open-source solutions are pretty good. But unless you have 10k compute nodes how can you bear the cost of having the expert knowledge necessary to debug this ?
Hardware sucks and is always on fire. Once you take this into account, the economic equation changes dramatically.
Re: Rolling your own servers with Kubernetes
#145Re: Rolling your own servers with Kubernetes
#146Earlier quoted context omitted.
I worked in a shop that had dedicated lab equipment and resources. Suddenly buying new equipment that we would use for years became like pulling teeth, no money they said. They couldn't explain what changed. Then just as suddenly they were willing to approve spending more than our one time equipment purchase... per month in monthly cloud costs / some poorly thought out cloud deployments.... Result: -1 Dedicated perso…
Sure, but the accountants got to move it from capEx to opEx and then everyone went out for a beer.
And the boss get to control a bigger budget, instead of having to ask for the money every time. It would probably be much nicer to work for a company where all exec and middle management where replaced by some AI mastermind, that would give compiler like answers:
- "sorry, we can't order this equipment because the inquiry does not end with a new-line character". Replace HR too, and the answer to your C++ job application would be "Sorry, we could not find c++ in your resume" (case sensitive, it has to be in lower-case). Think of how much money and grief that would save.
Re: Rolling your own servers with Kubernetes
#147Earlier quoted context omitted.
Sure, but the accountants got to move it from capEx to opEx and then everyone went out for a beer.
And the boss get to control a bigger budget, instead of having to ask for the money every time. It would probably be much nicer to work for a company where all exec and middle management where replaced by some AI mastermind, that would give compiler like answers: - "sorry, we can't order this equipment because the inquiry does not end with a new-line character". Replace HR too, and the answer to your C++ job applicat…
Add's c++ to resume.
"job added!"
This is awesome!
Re: Rolling your own servers with Kubernetes
#148Earlier quoted context omitted.
netflix does not depend on aws to deliver their video bits. they do it themselves based on a big network of core and edge pni and caches. this infra, that netflix has built, in datacenters, is a big competitive advantage. it would be worse performance and much higher cost to do this over something like aws cloudfront.
>this infra, that netflix has built, in datacenters, Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1] The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc. [1] https://arstechnica.com/information-technology/2015/08/netfl...
That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".
Re: Rolling your own servers with Kubernetes
#149Earlier quoted context omitted.
>this infra, that netflix has built, in datacenters, Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1] The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc. [1] https://arstechnica.com/information-technology/2015/08/netfl...
That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".
>That's still in datacenters
The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters.
Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user accounts signup, monthly billing, and analytics, etc workloads are at AWS. And as the article mentions, even updating the cache on the Netflix appliances is coordinated through AWS. The combination of those strategies keeps them out of the "datacenter business" and let's them stay focused on their core competency of "video content".
Re: Rolling your own servers with Kubernetes
#150Earlier quoted context omitted.
It mostly goes by power not rack units, if you have 48U and only 15amps, you're not going to fill it up. 15A is not a lot.
So, $400/mo is overkill power wise. What would the bandwidth bill look like? How much of an up front investment is it to buy 48U worth of hardware? Seems expensive...
I'm spending $375/mo for 8U, 12A, and 300mbit unlimited (3 x 100mbit bonded). The 8U gives me a GBe private network, 120 cores or so, 350GB RAM, and 20TB or so of storage.
The servers cost me $1200-$2200 each and I would buy a new one every quarter until I got to where I am today. I haven't added new hardware for almost 2 years, but its more than capable of handling my workloads (mostly web apps, lots of servers satisfying sql,ldap,etc dependencies). I burst (and failover) to all the clouds today.