Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving The New York Times Games Platform to Google App Engine

open.nytimes.com

Post reply on HN
Print view View original

61–70 of 152 posts

Re: Moving The New York Times Games Platform to Google App Engine

#61

Post by bastawhiz »

mattbillenstein wrote: post #46
vs2 wrote: post #2

"Due to the inelastic architecture of our AWS system, we needed to have the systems scaled up to handle our peak traffic at 10PM when the daily puzzle is published." WT... I had to reread this to make sure I didnt misunderstand... why not work on making the current arhictecture elastic?! #cloudPorn

I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.

That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.

Re: Moving The New York Times Games Platform to Google App Engine

#62

Post by yeukhon »

jsudhams wrote: post #58
nrjames wrote: post #36

I migrated a big data stack to GCP from AWS. Reasons: GCP has better documentation, the AWS console and various services confuse the heck out of me (I guess I'm getting too old), and the security integration between GCP services saves a huge amount of time. It's super easy and very fast to used the Google Compute Engine VMs. Given that the company I work for uses G Suite, it's a piece of cake to implement SSO and oth…

Wait until GCP becomes older with bigquery and other versions. Also GCP does not have regions ready and it takes forever to get a new region. Waiting for Mumbai region for more than 3 to 4 months now. But one thing i like about GCP is that it allows to the limit setting in terms of cost and ensure you wont cross it. In case of AWS it can give alerts but for some reason say all you team in in one location and there is…

AWS doesn't add a new region every 3-4 months either. Adding a new region is very complicated. Normally vendor does not actually build the DC themselves. They would source from other data centers in the region whenever possible. Building a new DC is not something can be taken lightly. Then finally local laws.

Re: Moving The New York Times Games Platform to Google App Engine

#64

Post by mattbillenstein »

bastawhiz wrote: post #61
mattbillenstein wrote: post #46

Earlier quoted context omitted.

I'm not sure why people are so enamored with ELB -- just terminate SSL at your web boxes using nginx and publish the public ips of all of these machines in your DNS records. You remove a bunch of ELB per-request costs doing it this way and you can scale it however you see fit.

That works great until a machine starts failing health checks and you need to take it out of rotation ASAP. It's also the case that DNS gets cached and not all users create the same load: one user making many requests will burden one server instead of having the load evenly distributed.

Clients will try another ip if they can't connect -- partial failures may be a problem, but in my experience as long as nginx is alive, you can load balance to a different web backend if the app processes on that machine are wedged.

I've deployed this solution in 50k req/s environments and not seen a single user be a problem like you mention -- any motivated bad actor could cause problems in either scenario I expect.

Re: Moving The New York Times Games Platform to Google App Engine

#67

Post by NightlyDev »

This doesn't really make much sense to me. How many peak users are there? What's the number of requests per second?

I can't imagine that the load would be so high that it wouldn't be possible to do it without GCP with three developers.

It would be way more interesting with performance details. :)

Re: Moving The New York Times Games Platform to Google App Engine

#68

Post by brianwawok »

tarr11 wrote: post #22
bsaul wrote: post #12

Has anybody had successful experience deploying docker containers on appengine ? Last time i tried, i had such a bad experience in terms of deployment speed ( time to build the image, then upload it, then waiting for the stuff to deploy) that i reverted to managing my own gce instance. But maybe i had bad luck..

I just tried to move a rails app to appengine. It uses the flexible environment (eg, docker). It took like 10 or 15 minutes to deploy, each time. Heroku does my deploy in about 5 mins. https://groups.google.com/forum/#!topic/google-appengine/hZM... GCloud is cheaper though. Also, VMs spin up in GCloud amazingly fast. Like 5 seconds. Feels like somebody at Gcloud just needs to go and fix this. No reason this is so bad…

Deploy is 15 seconds on GKE. Wonder why app engine is so bad.

Re: Moving The New York Times Games Platform to Google App Engine

#69

Post by vira28 »

obulpathi wrote: post #13

Two things keep coming up while comparing GCP and AWS: * This accomplishment would not have been possible for our three-person team of engineers with out Google Cloud (AWS is too low level, hard to work with and does not scale well). * We’ve also managed to cut our infrastructure costs in half during this time period (Per minute billing, seamless autoscaling, performance, sustained usage discounts, ... )

This thread here came at the right time. Today whole day I attended the DynamoDB training. Honestly, one thing that I understood is its cost based on reads and writes per second. Irrespective of the amount of read data per operation (whether its 1 bytes or 100 bytes), its always charged for 1KB. So, as a work around what they suggested is using a Kinesis, a Lambda and an another service to make the write operation as a batch, in such a way the reads are near 1KB always. He pitched it like thats the perfect way to do. The problem I see is too many moving pieces for a simple thing to achieve. If the Dynamo team makes the reads cost based on the actual data we are all set.

Re: Moving The New York Times Games Platform to Google App Engine

#70

Post by sdenton4 »

bastawhiz wrote: post #59
tziki wrote: post #17

Earlier quoted context omitted.

Serverless is probably far too immature for companies like NYT.

I don't disagree with you, but it's also true that you don't have to go all-in. I run services that run server-ful-ly on EC2, while requests to the "front-end" are handled by Lambda. If you've got one or two very hot endpoints and the majority of your server load spike is thanks to the processing for those endpoints, moving just those endpoints to Lambda could give you reassurance that you don't have all of your eggs…

Three devs seems like a bit of a bottleneck... Why maintain two separate architectures for a single problem if there's another option?
Post reply on HN