Good writeup! Google really screws you when you are looking for 100G speeds, it's almost insulting. For example redundant 100G dedicated interconnects are about $35K per month and that doesn't include VLAN attachments, colo x-connect fees, transit, etc. Not only that, they max out on 50G for VLAN attachments. To put this cost into perspective, you can buy two brand new 32 port 100G switches from Arista for the same a…
So you want to build your own data center
231–240 of 287 posts
Re: So you want to build your own data center
#232Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…
Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.
Yikes, that escalated quickly. I'm glad you escaped the Switch Grim Reaper and my condolences to the families of the rest :(
Re: So you want to build your own data center
#233I am just fascinated by the need of Datacenter. The scale is beyond comprehension. 10 years ago, before the word HyperScaler was even invented or popularised, I would have thought DC market to be on the decline or levelled off now or around this time. One reason being HyperScaler, AWS, Google, Microsoft, Meta, Apple, Tencent, Alibaba, to smaller ones like Oracle and IBM. They would all have their own DC, taking on mu…
I remember 30 years ago most people used one single computer for the family at home, half of the people i knew didn't have proper internet access. (and this is from a western perspective, the rest of the world was even far less digitized).
Now look at how much networked computers are around you. - your phone - one or multiple TV's - laptops/desktops etc - smart home appliances.
and this is just looking at the very small sample size of a normal household, add up to that things like the digitizalisation of factories, the digitalization of the rest of the world. (internet access has grown massively in the past 15 years in the developing world).
We have far more computers then a decade ago, and far more people have them aswell, and it shows very little signs of stopping.
IPv6 for instance, supports an absolutely infanthomable IP address space. (which people often seem to think is overkill), but looking at the past growth, i think having suchs a large address space is a wise choice.
Another thing which people seem to not notice is that a lot of older DC's are being phased out, mainly because these facilities are repurposed telephone exchanges and far less suitable for more power hungry computing power.
Re: So you want to build your own data center
#234Re: So you want to build your own data center
#235Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…
> Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time. Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.
Con: interacting with internal stakeholders is waaaaay different from doing the same for the general public paying you. See also: every mention of GCP that ever shows up in these threads
Plus all their SDKs would be written in php :-P
Re: So you want to build your own data center
#236Earlier quoted context omitted.
AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend) I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr. Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “…
My experience with AWS support has been downright freaky. With other vendors , when I call a support line with an obscure issue that maybe only I hit in the whole world I fully expect to explain it to an overseas call centre drone with a poor voice line and rudimentary English. Then I expect to have to repeatedly escalate over months and be told “We can’t reproduce this glaringly obvious bug, closed.” That’s ignoring…
Re: So you want to build your own data center
#237Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…
Many years ago I had a BlackDiamond dropped on my foot during installation at INTX LON1 for LINX, disabling me for hours. The switch in question was evidently cursed: later that week a spanning tree misconfiguration on the same unit then disabled LINX for hours, throwing half of Britain's ISP peering into temporary chaos, and everyone else involved in that project was dead within two years.
wait, what?
Re: So you want to build your own data center
#238Man, I get an anxiety attack just thinking about making this stuff work. Kudos to all the people doing this.
Re: So you want to build your own data center
#239@railway What would you say are your biggest threats? Power seems to the big one, especially when the AI power and electric vehicle demand will drive up kWh prices. Networking seems another one. I'm out of the loop, but it seems to me like the internet is still stuck at 2010 network capacity concepts like "10Gb". If networking had progressed as compute power has (e.g. NVMe disks can provide 25GB/s), 100Gb would be th…
High end network interfaces are entering the 800Gbps interface era right now.
also, in 2010 10Gbps network connectivity to end hosts was NOT common. (it was common for router uplinks and interconnects though.
Network interfaces have not scaled as nicely because getting fast enough lasers to handle higher then 100Gbps has been a challenge, and getting to higher speeds basically means doing wave division multiplex over multiple channels across a single fiber.
Also, density of connections per fiber has increased massivly because the cost of DWDM equipment has come down significantly.
Re: So you want to build your own data center
#240Earlier quoted context omitted.
FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!
Yeah. Cloud providers are the worst. Their egress costs moved from "expensive but not unreasonable" circa 2010, to "what the fuck" territory now. A 10G port should be in the range of $2k per month, I believe? I don't mind paying that much.