Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…
We had a bird land on a transformer up on a pole and blew fuses. A couple years later, I toured the facility and the fried carcass was still there on the ground below it.
So you want to build your own data center
221–230 of 287 posts
Re: So you want to build your own data center
#222Re: So you want to build your own data center
#223I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…
My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?
When I was at AWS, our team used to (religiously / proactively) keep track of customers having multiple complaints, especially repeat complaints (all of which manifested in to some form of downtime for them). Regardless of their spend, these customers ended up getting the "white glove" treatment, which otherwise is reserved for (potential) top spenders (though, engs are mostly oblivious to the numbers).
This is besides the fact that some account managers & support engs may indeed escalate (quite easily at that) to push product eng teams to really & immediately pay that tech debt that's hurting their customers.
Re: So you want to build your own data center
#224I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…
My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?
Re: So you want to build your own data center
#225Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but t…
Am a bit surprised Meta doesn't offer a cloud provider yet to compete with AWS/GCP. Especially considering how much R&D they've put into their infra.
Re: So you want to build your own data center
#226I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see…
My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?
2. More
3. AWS has this idea of “customer obsession.” They will spend an absurd amount of time trying to understand your business and make sense of it.
Re: So you want to build your own data center
#227Earlier quoted context omitted.
My AWS account manager took me fishing. That’s what you get for a >$1M/yr spend. I don’t sense they would roll in mud with me, which is kind of incredible. I wonder how much you need to spend to get into mud rolling territory?
AWS support in general is extremely good in my experience. (We pay for whatever the tier below Enterprise is called, I think it costs 10% of your spend) I’ve been on 4 hour screenshare with AWS engineers working through some infrastructure issues in the past, and we only spend $100k/yr. Even at the $100k/yr spend level, AWS regularly reaches out with offers to try new services they’re launching for free. We’ve said “…
With other vendors, when I call a support line with an obscure issue that maybe only I hit in the whole world I fully expect to explain it to an overseas call centre drone with a poor voice line and rudimentary English. Then I expect to have to repeatedly escalate over months and be told “We can’t reproduce this glaringly obvious bug, closed.” That’s ignoring the series of very closely related family of issues I dug up in the process of troubleshooting. Which they continue to ignore because it’s “out of scope” for the ticket. “Open a new ticket and go through the pain again, peasant!”
With AWS my experience has always been “I’ve fixed that right up for you, is there anything else you’d like help with?”. Often after mere minutes!
I’m usually left speechless, ready to go on a well-practiced tirade of how “We’re spending millions on this crap and none of it works properly!”, but instead I just sit there gawping like a fish out of water, stammer “No, thank you, that was it.” and hang up in shame.
I just don’t understand why no other enterprise on Earth seems to have support this good. What’s the thing holding them back!? Maybe they assume that good support works only for this tiny upstart org called Amazon that will clearly never amount to anything!
Re: So you want to build your own data center
#228Earlier quoted context omitted.
Same feeling here. It's like they wanted a way to "play datacenter in the browser", but then asked 30 different teams to do it on their own, and only have them come together after they are all done to put the pieces together. Then find out it's not good at all and go "oh well, I guess we'll polish it over in the UI" (not knowing that no serious scale works with a UI). If I can't have AWS I'll make do with GCP. But if…
Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.
One of the most bizarre things is the crazy bad resource hierarchy. There are multiple overlapping and incompatible ones. Resources, networks, storage, IAM, billing and org, none of it in a single universal hierarchy. It seems to mirror the idiosyncrasies of legacy enterprise organisations with their fiefdoms, instead of a cloud.
The next useless thing is how you just cannot use what you need when you need it in whatever way you want it. Almost all services are hyper-segmented requiring various premium tiers instead of them being universally available.
I get it, it's a great way to bundle things people don't want and extract as much money out of them, but that only really works if people have no alternative. And those two form the bad architecture/bad technology trifecta with this third one: a lot of services, maybe most of them, seem like some sort of 2005 model where a resource is backed by nothing more than some random managed VM in the backend, with all the problems (failure modes, inconsistent behaviour etc) that come with that model.
Perhaps the reason for those things is simple: Microsoft wanted a way to extract more money from their customers and lock them in even more. Moving workloads to Azure meant something different for them than it did for the rest of the world: you used to have a lage army of common windows sysadmin jobs where there was a lot of local control and local management loops, but when you move that to a common template in someone else's datacenter (Azure, essentially) you can ditch most of those loops and people. Granted, they created those local controls/loops themselves to get a school-to-work microsoft client pipeline (same as say, Cisco or oracle), but I doubt there is any new markets to cater to in that way. Since people tend to be the most expensive and most risky part of a business, being able to take more of them out of the loop, making more of them optional or making them more expendable/remote is a (short-term) positive thing in the spreadsheets of most MBAs, which is who most large companies cater to after all. This did of course backfire and we now have the same quantity of jobs but instead of sysadmin you get 'azure engineer' which is more of a crossover between operational helpdesk and technical application manager. But everyone wins: during your exodus you can sell it as modernisation, when you remove that on-prem burden you can shift your CAPEX and OPEX around, your quarter looks better when you can reduce headcount, and once your bonus is in, you can put some job postings out for the people you are now missing.
Technology-wise, the only thing that really changed was the ways in which people could cut corners. Some corners are pre-cut, while others are uncuttable. API-wise, it's a crapshoot, a turd attempted to be polished by a webui that hides the maelstrom of questionable residue below the surface.
Re: So you want to build your own data center
#229Earlier quoted context omitted.
Same feeling here. It's like they wanted a way to "play datacenter in the browser", but then asked 30 different teams to do it on their own, and only have them come together after they are all done to put the pieces together. Then find out it's not good at all and go "oh well, I guess we'll polish it over in the UI" (not knowing that no serious scale works with a UI). If I can't have AWS I'll make do with GCP. But if…
Seriously wondering what you guys experienced with Azure. Never had an issue and prefer it over AWS.
Re: So you want to build your own data center
#230Earlier quoted context omitted.
Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks. With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc Super exciting
How do bandwidth costs work now? do you pay the ISPs a flat fee, or is it still usage-based? how much cheaper is it compared to cloud providers?
It depends a lot on what kind of connection you require, and things like oversubscription and congestion control also come into play.
Peering ports with IXP's are usually flat rate, while ports in datacenters to end customers usually have more complex constructs.
Hyperscaler bandwith is notoriously expensive. for instance, a 100Gbps port on the AMS-IX is 2500$[1].
Now, you need to account for extra costs to actually use this port (some IP space, ASN number etc) but even with all that added up i think you will not get much more expensive then 400$ per month averaged in total over a year.
Now what makes comparing difficult is that hyperscalers are not transparant when it comes to connectivity costs. Looking at egress fees for example:
AWS seems to charge 1 cent per Gigabyte transferred for egress fees.
If we send data at line rate across our 100Gbps for an entire month we get the following:
100gbps = 12.5Gigabytes per second. 12.5 * 2 629 743 83 (number of seconds in a month) = 32871797875 Gigabytes 32871797875 / 0,001$ = 3.287.179,7875
thats 3,2 million dollars... compared to roughly 4000!
AWS also seems to offer "dedicated connections". at roughly 22 dollars per hour [3] (no clue if this is even comparble to an IXP port, but the comparison would still be fun to make).
22$ x 720 (hours per month = 15.840$, or roughly 3times the IXP port price.
In both cases, you are getting absolutely shafted by egress prices at cloud providers compared to doing it yourself.
[1] https://www.ams-ix.net/ams/pricing-us [2] https://aws.amazon.com/ec2/pricing/on-demand/ [3] https://aws.amazon.com/directconnect/pricing/