Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

arador.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#181

Post by zurn »

mikiem wrote: post #152

As a provider of IaaS Cloud and of dedicated servers and colo, I hear this argument all the time. No one ever seems to include the Network Engineers, monitoring systems, the routers (better have more than 1!), the switches (distribution and access layers), the maintenance, software licenses (where applicable), customer support, cost of IP addresses, Account Payable, ARIN membership, RADB membership, cross-connects, o…

The article compares cloud costs to putting a little x86 server on a colo-provided 100 Mbps connection. I think you are not talking about the same kind of service, and also this reference price does include amortized cost of network engineering.

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#183

Post by kev009 »

mikiem wrote: post #152

As a provider of IaaS Cloud and of dedicated servers and colo, I hear this argument all the time. No one ever seems to include the Network Engineers, monitoring systems, the routers (better have more than 1!), the switches (distribution and access layers), the maintenance, software licenses (where applicable), customer support, cost of IP addresses, Account Payable, ARIN membership, RADB membership, cross-connects, o…

Straw man argument, you're looking at around $300/month for 20A, 10+ rack units, 100mbit commit to play with in a lot of decent US facilities. That's a few pathetically provisioned VMs on most clouds. I have such a colo for my personal use. This doesn't exclude me from using other providers for DNS and MX slaving, or things like object storage (i.e. I use Tarsnap for DR) and there isn't really any overlap or lost efficiency in mixing and matching those higher level services. My facility's house bandwidth blend provides free DoS mitigation as a nice bonus.

Most facilities and providers don't require you to have your own ARIN registration and block. In fact it's quite difficult to get even the smallest blocks right now. Instead, an ISP will lease you IPs usually at around $1/month.

A pair of x86-64 boxes running OpenBSD can easily virtual switch, route and firewall well above 1gbps. They can announce BGP if needed, if you have a delegation or are muli-homing. A pair of high quality 10g switches may push a few grand on the secondary market but you can ~get away~ with Ubiquiti gear.

This kind of setup can scale up pretty high by dropping the OpenBSD boxes from the forwarding plane and just using them as route reflectors into higher end switches.

There is a slight labor disadvantage that amortizes quickly. i.e. setting up switches and routers isn't hard, and if you aren't sysadmin skilled enough to do that in a few days you probably shouldn't be running VM infrastructure either and retreat to something like Heroku, Lambda or GAE where there is much less room for footshots.

So TL;DR $300/mo and 5k CapEx will run a company, double or triple that for DR of a serious startup. Reap significant saving as a larger company by partnering or outsourcing remote hands and basic ops with a company like yours for the day to day stuff.

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#184

Post by slackingoff2017 »

Spooky23 wrote: post #144
slackingoff2017 wrote: post #92

Earlier quoted context omitted.

I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and CGI was all there was, even then uptime was so good that it was rare to hit a website that was down. There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard t…

You're comparing the simplist possible scenario with a platform that can replace a large data center. How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs? The answer historically is that you would buy a bunch of stuf…

You don't need anything but regular PC's these days... 2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap.

Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.

Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.

Redundant transit handled by datacenter level multihoming.

Extra redundancy using dual data centers if you want with DNS round robin.

Buy dedicated lines to avoid bandwidth costs.

You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.

Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.

Now everybody can have a great setup as long as they pay out the nose.

Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.

You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#185

Post by overcyn »

throwaway-1209 wrote: post #182

Amazing to see the number of people who try hard to justify this blatant rip off pricing. This is coming from the same group of people who complain endlessly about the cost of wireless data and telco data caps.

With telcos, there are only a few companies to choose from. If you want to run your own dedicated servers, nothing is stopping you. So what if people want to pay more for the ease of cloud platforms?

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#186

Post by slackingoff2017 »

carterehsmith wrote: post #117
slackingoff2017 wrote: post #11

How is this is a surprise to anyone? The big players are all pushing their clouds because its a cash bonanza. It's the SaaS model for hardware, make money forever because your customers never own anything. I've done the math many times and it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.

> it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware. It may be. But for many people, it does not matter. Let's say you work at the place (I do) where "compute" cost is like ~0.5% of the total costs. Almost all of the cost is employee's salaries and benefits. BTW we get (and from what I hear, everyone does) offers from other cloud vendors to switch, and g…

Fair enough. I just think that infastructure is a bigger differentiator of success than most give it credit for. By using the cloud you're giving up the possible performance and money advantages of doing it yourself.

It's really just a new form of vertical integration. If you're working only at the high levels you're not only paying someone to do that rest for you, those people are making profit off of you. Profit that could be yours.

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#187

Post by MBCook »

rdtsc wrote: post #136
MBCook wrote: post #71

Earlier quoted context omitted.

#3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.

Is sabotage something you are worried about in general. Just seems like there might be deeper and serious issues if that is a primary​ concern.

It was sabotage in the 'oops this project isn't working' or 'oops this really import thing came up, no time' or 'didn't I do that?' sense. Having some plausible deniability. Work slowdown/stoppage stuff. Not actively breaking things (mostly). And it was all directed at maintaining control/power.

Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#188

Post by kernelsanderz »

I would imagine contention ratios would also factor into pricing. At least in Australia you might have a 8:1 ratio of actual bandwidth available for a consumer plan, and 3:1 for a business plan. I'd imagine that you pretty much get all the bandwidth you pay for in a data centre. I've certainly saturated 100mbps connections on servers in the past, 24x7. But perhaps people more knowledgeable than me could comment on this?

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#189

Post by MBCook »

Spooky23 wrote: post #146
_jal wrote: post #103

Earlier quoted context omitted.

...That sounds more like an HR problem than a technical problem. There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.

There are, but they are hard to find. Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else. I've run into awesome network guys in position of power…

We saw some good people, but they generally left because they didn't want to work in that environment.

Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.

Re: Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing

#190

Post by slackingoff2017 »

mbesto wrote: post #111
slackingoff2017 wrote: post #92

Earlier quoted context omitted.

I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and CGI was all there was, even then uptime was so good that it was rare to hit a website that was down. There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard t…

> Before all these cloud providers websites were not noticeably less reliable than they are now. Before all of these providers existed: 1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructur…

As we've gotten more device computers have gotten more powerful. Server software has gotten better. HTTP has Keepalive and multiplexing now. Encryption and networking are offloaded to hardware and we have a lot more cores.

I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets.

Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly.

I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds.

Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.

Post reply on HN