Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

reddit.com

Post reply on HN
Print view View original

111–120 of 157 posts

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#111

Post by SkyMarshal »

krobertson wrote: post #93
watchandwait wrote: post #59

Earlier quoted context omitted.

My experience with the AWS RDS database product has been excellent.

We looked at RDS and had a call with some of their engineers, but we basically had our EC2 + raid'd EBS set up almost the same as they did, all best practices already being done. Since RDS really is EC2 + EBS, they couldn't provide any real assurances it performed better than our own installation. We ended up moving off of AWS as a whole. After several discussions about how we can continue to scale, the ultimate answ…

Did you guys find a better cloud service, or did you roll your own in a datacenter somewhere?

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#112

Post by krobertson »

SkyMarshal wrote: post #111
krobertson wrote: post #93

Earlier quoted context omitted.

We looked at RDS and had a call with some of their engineers, but we basically had our EC2 + raid'd EBS set up almost the same as they did, all best practices already being done. Since RDS really is EC2 + EBS, they couldn't provide any real assurances it performed better than our own installation. We ended up moving off of AWS as a whole. After several discussions about how we can continue to scale, the ultimate answ…

Did you guys find a better cloud service, or did you roll your own in a datacenter somewhere?

We went with a managed hosting provider who built us a private cluster. Basically a private cloud. But that way we could get a dedicated SAN and move our DB servers out to dedicated boxes with whatever disk configuration we desired.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#114

Post by delano »

bretpiatt wrote: post #108
delano wrote: post #90

Earlier quoted context omitted.

If you're interested to see how sites perform on EC2 and Rackspace over time: https://www.blamestella.com/vendor/ec2 https://www.blamestella.com/vendor/rackspace

You're monitoring from AWS US-East it looks like, you'll want to mention that to give people some context around the latency numbers.

That true but I think what's more interesting is the number of incidents (timeouts, exceptions, and significant slowdowns).

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#115

Post by WALoeIII »

ceejayoz wrote: post #74
WALoeIII wrote: post #69

Earlier quoted context omitted.

You can't do multi-master with MySQL anyways, which until very recently has been the only "engine" RDS supports. Even if you could do multi-master, replication is still single threaded. You have to come up with your own sharding scheme. This is a limitation of MySQL not RDS.

You can't do multi-master with MySQL? News to me - we've been using circular replication between two servers, each a master and slave, for quite some time now. Not possible with RDS, unfortunately, but works fine on two EC2 instances.

There is a difference between multi-master and circular replication. To me, Multi-master is that I can write to both masters at the same time, which implies there is a way to resolve conflicts. Databases like Cassandra (timestamps) and Riak (vector clocks) have this, MySQL does not. If you write to the same record on both masters bad shit happens and its very hard to sort out.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#116

Post by hn_throw_away »

jedsmith wrote: post #14
jameskilton wrote: post #6

This comment further down, supposedly from an Amazon employee, paints a grim picture for EBS: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

The perspectives of disgruntled employees have been known to be worse than reality, on occasion. Not definitively saying that's the case here, just saying.

I work at Amazon - a lot of teams are like this. They're stuck managing a woefully broken product and spend all of their time propping up the beast, leaving no capacity left for meaningful fixes (in these cases, meaningful fixes are always gigantic engineering projects).

The team develops a reputation internally for being glorified firefighting, and have trouble recruiting. More senior engineers eventually flee (having, well, choice in the matter) leaving a team heavy with junior talent with no seasoned gurus leading the way.

The company is also growing at ludicrous speed, and hiring is difficult. When the product is in such a painful state, attrition from the team is high, and with slow hiring you are barely countering attrition (exacerbating the junior talent problem), and not even close to growing the team to be in a position to take care of the problem for good.

I suspect this is an industry-wide problem though, and is hardly unique to this place.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#117

Post by jread »

I was at the Cloud Connect conference last week. In a session on cloud performance Adrian Cockcroft (Netflix's Cloud Architect) spoke and said they do not use EBS for performance and reliability issues. They initially had some bad experiences with EBS and because of this decided to stick with ephemeral storage almost exclusively.

The guys from Reddit also spoke about their use of EC2. Apparently they are running entirely on m1 instances which suffer from notoriously poor EBS performance relative to m2 and cc1/cg1 instances.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#118

Post by absconditus »

rbranson wrote: post #107
danielrhodes wrote: post #88

Earlier quoted context omitted.

Amazon.com is not hosted on EC2. It's entirely separate.

This isn't entirely true. Amazon.com uses EC2 in addition to dedicated servers. http://searchcloudcomputing.techtarget.com/news/1516269/Amaz...

"She's the director of IT services for the retail giant, although she has nothing to do with the main website operations"

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#119

Post by fuzzmeister »

floodfx wrote: post #75

I'll probably be downvoted for this but seems to me the root cause of this problem is Reddit's architectural decision to remain in a single availability zone. If it wasn't EBS it could have been some other issue related to the single AZ that could have brought the site down. Blaming EBS, particularly if you knew it to be a potential weakness in your architecture, seems like a deflection of responsibility.

Exactly. While Amazon clearly tries to make single-zone reliability as good as possible, I think they expect customers to use a multi-AZ setup if they expect true reliability.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#120

Post by fuzzmeister »

ck2 wrote: post #24

I firmly believe "the cloud" is a fad, unless for some reason you own and operate all the hardware yourself (ie. Google). Like other technical fads, everyone will probably come back to servers they can reach out and touch when needed, sooner or later.

That is entirely wrong. With AWS, we've built a multi-AZ load balanced infrastructure for very little time and money. Getting an equivalent setup out of our own hardware would have been orders of magnitude more expensive and time consuming.
Post reply on HN