"Amazon's EBSs are a barrel of laughs in terms of performance and reliability" • Hacker News Board

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#141

Post by jodrellblank » Sat, Mar 19, 2011, 4:00 AM UTC

j_s wrote: post #140

Being totally new to AWS, why does everyone skip right past using ZFS? http://blogs.sun.com/marchamilton/entry/a_brilliant_argument... "Cloud Storage Will Be Limited By Drive Reliability, Bandwidth ... The key feature of ZFS enabling data integrity is the 256-bit checksum that protects your data."

ZFS will ensure that what was written to disk comes back to memory consistently, or with errors spotted. It wont ensure that the right thing was written to disk, or that the database IDs which were written leave your database relationships in a consistent state, etc.

ZFS will do nothing about this "More recently we also discovered that these disks will also frequently report that a disk transaction has been committed to hardware but are flat-out lying.", for instance, other than tell you the data you want isn't there to be read - like any filesystem would.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#142

Post by bgentry » Sat, Mar 19, 2011, 4:08 AM UTC

hn_throw_away wrote: post #116

jedsmith wrote: post #14

Earlier quoted context omitted.

The perspectives of disgruntled employees have been known to be worse than reality, on occasion. Not definitively saying that's the case here, just saying.

I work at Amazon - a lot of teams are like this. They're stuck managing a woefully broken product and spend all of their time propping up the beast, leaving no capacity left for meaningful fixes (in these cases, meaningful fixes are always gigantic engineering projects). The team develops a reputation internally for being glorified firefighting, and have trouble recruiting. More senior engineers eventually flee (havi…

I wonder if it's time for AWS to open a development office in a more startup-oriented city (i.e. SF). It might help them attract and retain more talent.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#143

Post by callmevlad » Sat, Mar 19, 2011, 6:03 AM UTC

hn_throw_away wrote: post #116

jedsmith wrote: post #14

Earlier quoted context omitted.

The perspectives of disgruntled employees have been known to be worse than reality, on occasion. Not definitively saying that's the case here, just saying.

I work at Amazon - a lot of teams are like this. They're stuck managing a woefully broken product and spend all of their time propping up the beast, leaving no capacity left for meaningful fixes (in these cases, meaningful fixes are always gigantic engineering projects). The team develops a reputation internally for being glorified firefighting, and have trouble recruiting. More senior engineers eventually flee (havi…

This is quite disturbing to read about the place that tons of companies rely on to host their apps. As EC2 (and friends) grows, it's growing undoubtedly more complex. Combine that with many of the original/senior devs leaving to greener pastures and seems like as more time passes it becomes more risky to host with Amazon. Doesn't really inspire confidence ...

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#144

Post by gexla » Sat, Mar 19, 2011, 6:04 AM UTC

jameskilton wrote: post #6

This comment further down, supposedly from an Amazon employee, paints a grim picture for EBS: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

Yeah, I call BS on this one. Amazon AWS is for certain use cases. It has never been a platform for all solutions. In fact, it has mostly been a platform for people to craft their own solutions. AWS is not a web hosting platform. If you want a web hosting platform, you create one the best you can from the tools available. This is the sort of response I would expect from an employee of AWS, not the one I saw in those comments. That or maybe the comment was from a customer service guy who isn't a developer.

I'm surprised Reddit ever though AWS would be a good platform to host on. You don't bitch about it, you create the best system you can and if something doesn't work, then you need to do more work. If you don't want to put in the work, then AWS is wrong for you. You don't see Heroku bitching about AWS, rather they made the thing work for them with great engineers.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#145

Post by blasdel » Sat, Mar 19, 2011, 7:00 AM UTC

powertower wrote: post #94

jedsmith wrote: post #30

Earlier quoted context omitted.

The cloud significantly lowers capital expenditure to get into an Internet-enabled business, which cultivates the very startup ecology that Y Combinator exists to leverage and support. Those teenagers who started the Facebook Pokemon game would have never had the resources to build a scalable solution with hardware that they own. (That is, unless Y Combinator paid a lot more money as part of participating. They might…

Cloud = Marketing(VPS);

It's not quite that simple — there's another level of 'true cloud' platform services like GAE, Heroku, Force.com, etc. that really deliver on the promise.

PaaS = Productize(VPS);

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#146

Post by WALoeIII » Sat, Mar 19, 2011, 7:37 AM UTC

ceejayoz wrote: post #128

WALoeIII wrote: post #115

Earlier quoted context omitted.

There is a difference between multi-master and circular replication. To me, Multi-master is that I can write to both masters at the same time, which implies there is a way to resolve conflicts. Databases like Cassandra (timestamps) and Riak (vector clocks) have this, MySQL does not. If you write to the same record on both masters bad shit happens and its very hard to sort out.

You can write to both masters at the same time in a MySQL multi-master circular replication setup. It's done via auto_increment_increment and auto_increment_offset configuration settings in my.cnf - each server generates autoincrement keys that are unique to that server.

This still doesn't protect you from UPDATE statements. I suppose you could pull it off if you could be absolutely certain that you were only CREATEing. You still have the problem that replication is single threaded, so this doesn't scale your writes beyond one thread.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#147

Post by WALoeIII » Sat, Mar 19, 2011, 7:46 AM UTC

bgentry wrote: post #142

hn_throw_away wrote: post #116

Earlier quoted context omitted.

I work at Amazon - a lot of teams are like this. They're stuck managing a woefully broken product and spend all of their time propping up the beast, leaving no capacity left for meaningful fixes (in these cases, meaningful fixes are always gigantic engineering projects). The team develops a reputation internally for being glorified firefighting, and have trouble recruiting. More senior engineers eventually flee (havi…

I wonder if it's time for AWS to open a development office in a more startup-oriented city (i.e. SF). It might help them attract and retain more talent.

A.) Amazon isn't a startup. B.) A lot of Bay Area companies (Zynga, Facebook, Salesforce) are opening Seattle offices to take advantage of the Amazon and Microsoft talent pools. C.) They already have a Bay Area office. http://public.a2z.com/index.html I believe some core SimpleDB guys (Jim Larson) were based out of there.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#148

Post by cpg » Sat, Mar 19, 2011, 10:46 AM UTC

This seems too much of a coincidence.

We released a dropbox-like product to sync and the back-end is on EBS. Yesterday we saw two times when a device got filled to 7GB and as it got closer it became slower and slower and slower. We did not have any instrumentation/monitoring in place and we were immediately suspect it was something on our end.

We (wrongly?) assumed reliability and (decent) performance from AWS.

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#149

Post by gaius » Sat, Mar 19, 2011, 2:17 PM UTC

ck2 wrote: post #24

I firmly believe "the cloud" is a fad, unless for some reason you own and operate all the hardware yourself (ie. Google). Like other technical fads, everyone will probably come back to servers they can reach out and touch when needed, sooner or later.

You know, back in the 70s, there was the concept of a "computer bureau", these would be some people who had a mainframe and you would rent time on it by the hour, so if you had a payroll run, or a simulation, or whatever, you would upload it to them via a modem (or courier them the punchcards!), run it there, download the results (or get them delivered printed out). Early BBSs and MUDs often ran in spare capacity on these mainframes.

There ain't nothin' new under the sun...

Re: "Amazon's EBSs are a barrel of laughs in terms of performance and reliability"

#150

Post by vdondeti » Sat, Mar 19, 2011, 3:50 PM UTC

fuzzmeister wrote: post #120

ck2 wrote: post #24

I firmly believe "the cloud" is a fad, unless for some reason you own and operate all the hardware yourself (ie. Google). Like other technical fads, everyone will probably come back to servers they can reach out and touch when needed, sooner or later.

That is entirely wrong. With AWS, we've built a multi-AZ load balanced infrastructure for very little time and money. Getting an equivalent setup out of our own hardware would have been orders of magnitude more expensive and time consuming.

Can you please explain how you built a multi-AZ load balanced infrastructure, given that Amazon's ELB only load balances within a given AZ. I assume you used some external service. Would you mind providing the details. Thanks.