Why Stack Exchange Isn’t in the Cloud • Hacker News Board

Re: Why Stack Exchange Isn’t in the Cloud

#81

Post by ssmoot » Fri, Nov 18, 2011, 8:27 PM UTC

reuser wrote: post #63

ssmoot wrote: post #52

Earlier quoted context omitted.

Instead of 12 physical cores, 96GB of RAM and a 2TB SSD Array pushing 1M IOPs on dedicated hardware for my PostgreSQL database servers, I'd need 1TB of RAM in an AWS box because I'll be lucky if I can even break 10K in IOPs. Does the price make sense then? I have yet to see any significant AWS deployment that doesn't feel like it could be done better, more reliably, and much more cheaply as a co-located setup.

You can't really create a massive co-located setup on demand for big jobs, then tear it down ... use the right tools for whatever job you're doing. EC2 being more expensive for your Postgres deploy doesn't mean it is an expensive toy

I don't think I could say it better: http://twitter.com/#!/dhh/status/133656040561577984

If you really have a need to setup and teardown a bunch of cores for the occassional large batch, nobody's questioning that the cloud is an economical way to do that.

But I've never had to do that. Not in a way that would make the development time involved economical anyway. Wait two hours for this once-in-a-blue-moon processing job to finish, or spend a day setting up a process to handle such jobs quickly in the future?

It'd really take something exceptional (again, with low IOPs demands) to have that make much sense unless I was already hosting in the cloud and had invested money in making such a task quick and cheap.

Re: Why Stack Exchange Isn’t in the Cloud

#82

Post by ssmoot » Fri, Nov 18, 2011, 8:43 PM UTC

nupark2 wrote: post #78

ssmoot wrote: post #52

Earlier quoted context omitted.

Instead of 12 physical cores, 96GB of RAM and a 2TB SSD Array pushing 1M IOPs on dedicated hardware for my PostgreSQL database servers, I'd need 1TB of RAM in an AWS box because I'll be lucky if I can even break 10K in IOPs. Does the price make sense then? I have yet to see any significant AWS deployment that doesn't feel like it could be done better, more reliably, and much more cheaply as a co-located setup.

Instead of 12 physical cores, 96GB of RAM and a 2TB SSD Array pushing 1M IOPs on dedicated hardware for my PostgreSQL database servers, I'd need 1TB of RAM in an AWS box because I'll be lucky if I can even break 10K in IOPs. Does the price make sense then? Does it? That's the point of analysis. It may be that you can scale up your organization on AWS, and then make the decision to scale something like a database vert…

> Does it?

No, it doesn't. You're throwing up a spectre. This are pretty easy numbers to come by. The point in this one example being vertical scaling options are pretty constrained on something designed to really only effectively scale cores and RAM.

> That's a stretch. Especially the "cheaply" part.

All this is a straw man. I really have a hard time believing "I don't firmly fall on either side of the debate". In fact, I call BS.

Who actually has to wire their own PDUs unless they want to? Or is forced to buy cages, racks, etc?

If you want to I suppose you can, but I haven't hit a Tier 1 data-center where that's even an option. Unless you were to buy an unfurnished cage perhaps on the terms you got to hire your own people for the build out.

Otherwise, for most deployments, even for something like Reddit, you're talking about stacking a couple 48 port switches, racking a few servers, and making sure you don't screw up airflow with bad cabling. Just spend $500 on an experienced cable guy to wire it up. Cabling isn't the most fun.

Staffing costs are a drop in the bucket. Racking a couple dozen systems, configuring your ports is pretty trivial.

If you have a business with dependable positive cash flow affording you the luxury of signing a three year hardware lease, it doesn't make cash sense to do anything else for 99% of deployments.

The "core competency" stuff is just a salve for developers who want to live in a homogenous environment. Life gets easier with a competent IT person. Not harder. You still have to monitor processes, setup syslog servers, archive logs, monitor disk space, load, available RAM, bandwidth, trace down abusers, setup mail servers for at least internal monitoring, create backup procedures, automate deployments, figure out how to compile some old library.

All of this stuff is where your IT staff effort goes. Not in the once-in-a-blue-moon-we-have-to-rack-some-servers tasks. You aren't better off because Amazon is handling your power requirements instead of a good colo. Those things are details you never have to think about either way. Saying so is a definite straw-man.

Re: Why Stack Exchange Isn’t in the Cloud

#83

Post by asharp » Sat, Nov 19, 2011, 12:36 PM UTC

ericd wrote: post #74

asharp wrote: post #70

Earlier quoted context omitted.

Saying that you can't get IOPS on AWS is like saying that your machinegun sucks at grating cheese. It's really not meant to do that. But that's fine as AWS isn't the only cloud platform out there, and some are actually desgined to take data intensive workloads.

The problem is that it's touted as a great way to scale web apps. Since DB performance is usually the limiting factor in webapp scaling, this doesn't appear to stack up. Needing a lot of IOPS is pretty par for the course.

For certain workloads it deals ok, but yeah, I'd fully agree with you.

On the other hand, there are plenty of clouds out there that have reasonable IOP performance, so it's not as though it's a monopoly.

Re: Why Stack Exchange Isn’t in the Cloud

#84

Post by asharp » Sat, Nov 19, 2011, 12:41 PM UTC

quanticle wrote: post #76

asharp wrote: post #72

Earlier quoted context omitted.

Keep in mind that screwing with syscalls is an AWS thing, not a virtualisation thing. They make fsync wierd for good reason though, and they really don't have much they can do.

Right. The question, however, was about working in the "cloud", which usually means that you just have access to the VMs, not to the underlying hardware. VMs are great, but there's nothing like root access to the physical computer to help you diagnose hardware issues.

Yes and no.

Remember the interface between the vm and hypervisor is standardised and it is running many other functionaly identical vms. Also remember that platform they are running on is usually homogeneous, very well maintained, monitored, and understood.

Also keep in mind that with 'hardware issues' you can reploy onto another machine in minutes, rather then having to wait for say a replacement part from dell/etc. Your standard procedures should come into play in both cases (Failing to a secondary server, bringing a tertiary server into standby, etc.)

Re: Why Stack Exchange Isn’t in the Cloud

#85

Post by ericd » Sun, Nov 20, 2011, 2:01 AM UTC

asharp wrote: post #83

ericd wrote: post #74

Earlier quoted context omitted.

The problem is that it's touted as a great way to scale web apps. Since DB performance is usually the limiting factor in webapp scaling, this doesn't appear to stack up. Needing a lot of IOPS is pretty par for the course.

For certain workloads it deals ok, but yeah, I'd fully agree with you. On the other hand, there are plenty of clouds out there that have reasonable IOP performance, so it's not as though it's a monopoly.

I guess my question is why go for reasonable when you can get incredible performance per dollar with RAID-10 SSDs? A few dedicated DB machines with SSDs and many cores can get you absolutely monstrous throughput without going for any of the more exotic DB solutions.