Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Obama Campaign AWS Infrastructure In One Diagram

awsofa.info

Post reply on HN
Print view View original

91–100 of 106 posts

Re: Obama Campaign AWS Infrastructure In One Diagram

#91

Post by Ogre »

miles932 wrote: post #84
Ogre wrote: post #38

Earlier quoted context omitted.

I tried to figure out what software this was from, but I gave up. I did find that Wikipedia has a nice list of network diagram software here http://en.wikipedia.org/wiki/Comparison_of_network_diagram_s... I'd love to describe a network in (something like) graphviz dot notation and get a diagram that looks like this out the other end. Usually I wind up just using Visio and doing it by hand because work pays for it any…

It's good 'ol Inkscape :)

Thanks for the answer! I was just playing with Inkscape the other day, I'll make a point of learning it for real next time I need to put something like this together.

Re: Obama Campaign AWS Infrastructure In One Diagram

#92

Post by atsaloli »

aasarava wrote: post #57
bm1362 wrote: post #39

Earlier quoted context omitted.

Texas public school, we studied government in the 12th year.

CA public school. Same.

Thank you. I dropped out in the 11th grade (went to college) which explains why I missed civics in the 12th. (I went to school in California.)

I caught up recently by drilling my Aussie wife for her US Citizen civics test - she learned all 100 questions by heart.

I made her Web flash cards, too: http://www.verticalsysadmin.com/US.html

I dropped out of college, too. Now I don't have an EDUCATION section on my resume. Plenty of work experience though.

Re: Obama Campaign AWS Infrastructure In One Diagram

#93

Post by jwilliams »

miles932 wrote: post #61
jwilliams wrote: post #33

Wish this had a legend. Obviously the box sizes and positioning mean something. What I can see makes me a bit curious. - Frontend Web boxes are "big", where the (some) of the backend App boxes are small. I'd have though it would be the reverse -- frontend horizontally scaled first, backend vertically scaled first. - Sometimes they use Zones ABC, sometimes AB. This is cool, but a lot of the "front" infrastructure is A…

Box sizes: this system supported lots of different programming languages/framewoerks/pre-built OSS, and as a result, some parts ran nice and lean (python) others chewed memory with reckless abandon (magento). Another factor was supplying enough network bandwidth to some hosts, hence larger sizes. Zones: most apps were built for 2 AZ's at the start; apps deemed "critical" and "doable" flipped to 3 near the election. A…

Cool - Thanks for the replies!

Re: Obama Campaign AWS Infrastructure In One Diagram

#94

Post by trotsky »

bonertruck wrote: post #77
bonertruck wrote: post #72

Earlier quoted context omitted.

The campaign isn't a tech startup. Its a people startup that uses tech to facilitate the things that people do.

Perhaps that was a bit overly broad and not quite representative of what I meant to convey. The problems faced by a campaign and the problems faced by a tech start up are inclusive but not all encompassing. Every dollar you save on your server infrastructure is a cell phone plan for a lonely field office in Texas struggling to make a dent in a hopeless battle but they're there because they care. Are you gonna tell th…

I guess I was being too dry, but I was suggesting that such massive deployments in multiple regions costs far more in ongoing fees and engineering effort than an equivalently performing environment that is managed by the end user (campaign). And yet with all that expense you still end up with less reliability and transient issues that you end up compensating for by even more over provisioning and rearchitecture.

AWS is good for organizations that need to minimize upfront costs, orgs too small to support suitable ops staff, overflow traffic, people who hope they're about to hockeystick growth or have no idea if they'll even be around in a year. Also it's good for people busy wasting limited partner's money.

It is, however, a pretty bad choice for a campaign with huge coffers, a large staff, certainty about the length of time they'll be in operation, and a very predictable seasonal use pattern. And of course they do it all with money from small contributors.

I'm not saying that aws or especially cloud models aren't a wonderful tool in a lot of cases. But we're training a whole generation of web jockies who are reflexively going to AWS because they think ops and infrastructure is scary.

Re: Obama Campaign AWS Infrastructure In One Diagram

#95

Post by parasubvert »

Really jazzed by CloudOpt, which I didn't know about. I had a multi-region architecture supporting AWS offshore development teams and HA between US-East & US-West, and this would have been a godsend. We had Windows Servers as dev workstations in Singapore and devs were based in India and Canada -- latency was actually fine to the desktop, and we used Amazon's internet connection to haul data back to US-East before hitting our data centre. This was satisfactory for most connectivity (throughput was fine) but occasionally the encapsulated routing would cause packet retry pains that traditional WAN acceleration may have helped with a bunch.

But, overall, this is great work, and shows what's possible to those in the enterprise that think that Amazon is still a bookseller, and clouds are basically things in the sky you don't want on your golf days. This will only help further the progression to way less wasted cost & time in IT. (Unless you're a republican, I suppose. Or maybe especially if you're a republican, you should be studying this for 2016 and considering an alternative to hiring Very Serious IT Consultants.)

Re: Obama Campaign AWS Infrastructure In One Diagram

#96

Post by scottvdp »

pfg wrote: post #47
jwilliams wrote: post #33

Wish this had a legend. Obviously the box sizes and positioning mean something. What I can see makes me a bit curious. - Frontend Web boxes are "big", where the (some) of the backend App boxes are small. I'd have though it would be the reverse -- frontend horizontally scaled first, backend vertically scaled first. - Sometimes they use Zones ABC, sometimes AB. This is cool, but a lot of the "front" infrastructure is A…

I remember reading somewhere that they were using Akamai as their CDN provider (at least for their donation page).

We used Akamai, Cloudfront, and Level3 in various combinations.

Re: Obama Campaign AWS Infrastructure In One Diagram

#97

Post by scottvdp »

mratzloff wrote: post #37

The most surprising thing about this is that test and staging are 1:1 duplicates of production. At the companies I've worked at dealing in scale, that has never been the case. "Oh, you run 10 servers and 2 load balancers across these 2 sub-systems? Here's 2 servers and a toy whistle for staging." But then again, I suppose that's what you get when you can dump hundreds of millions of dollars on a problem over a couple…

That part is a a little more nebulous. When deploying the entire full stack of all applications is automated, rolling a new environment is very simple. So, for instance, if I wanted to load test a specific end point in our API, I would just build a new instantiation of the API and test against that in a protected and isolated environment.

So in reality, it might be 2 staging environments and 12 testing. There was no real set number of environments.

Re: Obama Campaign AWS Infrastructure In One Diagram

#98

Post by scottvdp »

stevewilhelm wrote: post #36

I would really be interested in know a) if I had a copy of the Puppet scripts how long would it take get this server configuration up and running and b) how much would the AWS costs be per month?

A) That's a bit nuanced and depends on the application you are referring to (there are ~200 represented here), but if you had the scripts and the repos and built from a vanilla AMI, a box would boot and configure within a few minutes. We shortened the boot times greatly by using puppet to configure a template node which we then used to create a generic versioned AMI per autoscale environment and deployed with Asgard. If you had the AMIs, an EBS backed instance would boot and serve traffic in anywhere between 15 and 30 seconds or so.

B) For costs, you can check the FEC filings... I cannot recall what they were month to month, but it will be accurately reflected quarterly there, so you can get a rough estimate. It was like a kabillion times more in October 2012 than it was in June 2011. The important part for us is that the costs scaled with our demand.

Re: Obama Campaign AWS Infrastructure In One Diagram

#99

Post by scottvdp »

minimax wrote: post #21

Harper Reed and Scott VanDenPlas did a talk / Q&A at the Chicago AWS meetup group a few months ago. My hazy recollection of the salient points of the talk: 1) This represents a heterogenous set of services where the engineers in charge of each service were more-or-less free to use whatever technology stack they were most comfortable with. 2) Use puppet for provisioning everything. (There is some subtlety Scott went i…

The subtlety you are referring to in #2 is that we moved from puppet for deployments to puppet to configure template nodes. We baked AMIs from those template nodes and deployed those, drastically shortening our time from boot to serving traffic.

Re: Obama Campaign AWS Infrastructure In One Diagram

#100

Post by scottvdp »

ceejayoz wrote: post #87
dahumpty wrote: post #25

Its interesting to note that the the boweb-production servers of "www" don't have read-replicas (asynchronous replication) for read scalability. Anyone know why? Typically, one would use read-replicas to alleviate read workloads on the master.

Most likely the www servers are serving almost entirely static, cached content. Hell, they might not actually see any actual public traffic - they might just serve as CDN origins.

This is 100% correct. We also generated static snapshots of the origin and mirrored to S3. In the instance of something going awry with our primary origin servers, we could fail back to S3 as an origin.

EDIT: I'll add that our origin offload for WWW hovered around 98%.

Post reply on HN