121–130 of 157 posts
Why is reddit relying on only one cloud provider? AWS can/should do better but service providers of the size of reddit should be using mult-vendor set-ups for sure.
Earlier quoted context omitted.
Did you guys find a better cloud service, or did you roll your own in a datacenter somewhere?
We went with a managed hosting provider who built us a private cluster. Basically a private cloud. But that way we could get a dedicated SAN and move our DB servers out to dedicated boxes with whatever disk configuration we desired.
Earlier quoted context omitted.
This is totally true, but at the same time, given the success and scale of AWS, it's insane that they would not have the resources they need.
Word in the industry is that AWS is insanely profitable, so they've got not problem finding the money to hire the help. My gripe with EBS is that hiccups in EBS cause my Linux instances to "lock up", consume 100% CPU and become unresponsive. AMZN is providing their own Linux distribution and drivers for the EBS devices so they can also attack this problem by patching the Linux kernel.
Lesson for startups: start in the cloud, grow your business, build your own cloud. Never trust critical parts of your business to others.
Earlier quoted context omitted.
Amazon.com is not hosted on EC2. It's entirely separate.
This isn't entirely true. Amazon.com uses EC2 in addition to dedicated servers. http://searchcloudcomputing.techtarget.com/news/1516269/Amaz...
Earlier quoted context omitted.
If you understand the limitations of the various products you can build a VERY reliable service. The reddit assumption of a single datacenter and single technology to store that data was an engineering failure. They essentially didn't have a disaster recovery plan in place.
I'm sure reddit's engineers are as capable as any for producing a seemless disaster recovery plan, but the most common obstacle to implementing it is cost. Most web services choose the occasional risk of downtime in one data center instead of incurring the cost of being in two data centers at all times.
Earlier quoted context omitted.
One other huge downside of raiding EBS volumes is you can't use EBS's snapshotting features as you cannot guarantee a perfect sync (you could use LVM yourself however). Honestly, since EBS vols are supposedly not tied to a single disk, the raiding should be done on Amazon's end. That it isn't is telling.
You have to snapshot at the system level anyway if you want a consistent snapshot: otherwise the filesystem (or your database) could have been reordering and delaying writes that end up not being part of the "consistent snapshot". This is simply not a RAID-specific issue, nor is it a problem with EBS (as it is generally easy to use LVM, xfs, and/or PostgreSQL to handle that part of the job).
Why is all this necessary? If the system (i.e. DB + FS + block device) are all working as they should, then once a commit returns, the data should be on disk. If it's not, you have no guarantee data that you thought was committed will still be there after a kernel panic or power outage.
In that case, no amount of xfs-freeze or table flushing during a snapshot is going to save you from the fact that your DB is one kernel panic away from losing what the rest of your system believed were committed transactions.
Earlier quoted context omitted.
You can't do multi-master with MySQL? News to me - we've been using circular replication between two servers, each a master and slave, for quite some time now. Not possible with RDS, unfortunately, but works fine on two EC2 instances.
There is a difference between multi-master and circular replication. To me, Multi-master is that I can write to both masters at the same time, which implies there is a way to resolve conflicts. Databases like Cassandra (timestamps) and Riak (vector clocks) have this, MySQL does not. If you write to the same record on both masters bad shit happens and its very hard to sort out.
"Elastic" is AWS's claim to fame, but I am not seeing it.
Trying to resize an EMR cluster (which is half the point of having an EMR cluster instead of buying our own hardware) generates the cryptic error "Error: Cannot add instance groups to a master only job flow" that is not documented anywhere.
(Why would Amazon even implement a "master only job flow", which serves no purpose at all?)
This kind of complaint reminds me of people who buy a product that does A very well, but then they trash it in reviews for not doing B. It was never advertised as doing B, but you'd never know that from the complaining.