Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving The New York Times Games Platform to Google App Engine

open.nytimes.com

Post reply on HN
Print view View original

91–100 of 152 posts

Re: Moving The New York Times Games Platform to Google App Engine

#91

Post by manigandham »

zjaffee wrote: post #83
obulpathi wrote: post #30

Earlier quoted context omitted.

Adding more context. Sorry for missing it out in first place. I mostly work in Big Data Space. Google Clouds Big Data stuff is built for Streaming / Storing / Processing / Querying / Machine Learning at Internet Scale data (PubSub / Bigtable / Dataflow / BigQuery / Cloud ML). AWS scales to terabyte level loads. But, beyond that, its hard and super costly. Google's services autoscale to Petabyte levels / millions of u…

Hold on, please do not say google cloud scales well, yes they do have services that make a ton of claims, but unlike AWS, things don't work as promised which is magnified by the fact that their support is way worse. Additionally, Big Query is far more expensive than Athena, where you have to pay a huge premium on storage. The biggest difference is that what amazon provides you in infrastructure, where as google provi…

GCP support can sometimes be bad but these other claims don't add up. What isn't working as promised? BigQuery can do a lot more than Athena and it's storage pricing is the same or cheaper than S3.

We've used 5 different providers for a global system and GCP has won by both performance and price. We still use Azure and AWS for some missing functionality but the core services are much more solid and easy to deal with on GCP, which is also far more than just app engine.

Re: Moving The New York Times Games Platform to Google App Engine

#92

Post by funkjunky »

ledgerdev wrote: post #43
bsaul wrote: post #12

Has anybody had successful experience deploying docker containers on appengine ? Last time i tried, i had such a bad experience in terms of deployment speed ( time to build the image, then upload it, then waiting for the stuff to deploy) that i reverted to managing my own gce instance. But maybe i had bad luck..

Funny you mention that, I was just experiencing this yesterday in the flex environment. Re-deploy time for a 3 character text change on one page is 8+ minutes for each re-deploy. This is insane. Does anyone know how to speed that up?

(Google cloud support) Unfortunately, this is a pain point that has no easy solution at the moment. It doesn't matter how trivial a change is, it isn't the docker deployment that is taking all that time. As mentioned above, the bottleneck is updating the GC load balancer with the new routing rules, which takes time to propagate throughout the system. This is a high priority issue internally, but updating the load balancers is no trivial task, and will take a lot of time and testing.

In the meantime, I recommend the following mitigation strategies:

1. Try to get into the habit of carefully reviewing and testing new versions locally before deployment. Client libraries should still work if you have a valid default application credential set up. I say this because I have a hard time remembering to do this as well.

2. Static content and templates for your site should be hosted on GCS, not deployed with your app in a "static/" folder or something. Easy to fix typos, HTML/CSS, and JS errors by simply using gsutil to copy the fixed file over, takes only a second.

3. Always keep a stable version of your app available in case you broke something in a new deployment. It's quicker to route traffic to an older version than it is to track down a bug and wait for the fix to finish being deployed.

Not ideal, but again, Googlers have to suffer this too, and are very motivated to find a way to fix this.

Re: Moving The New York Times Games Platform to Google App Engine

#93

Post by ubernetes »

vira28 wrote: post #81
karmakaze wrote: post #79

Earlier quoted context omitted.

Why CloudSQL instead of Cloud Spanner? If for existing SQL workloads I can understand but for new services I'm admiring Spanner over DynamoDB

Looks like Spanner is a relational database. DDB is just a key-value pair. So, is it fair to compare against them or am i missing something?

Why wouldn't it be "fair"?

Re: Moving The New York Times Games Platform to Google App Engine

#94

Post by funkjunky »

bpicolo wrote: post #32
CobrastanJorji wrote: post #14

Earlier quoted context omitted.

Are you talking about this Python SDK doc? https://googlecloudplatform.github.io/google-cloud-python/la...

Yeah. In particular, I was really looking for something for "here's how to do most of the basics for Cloud Storage". If you look at https://googlecloudplatform.github.io/google-cloud-python/la... , for example, there's not even a top level navigation index that I can read through to guess what function I might need by name.

(Google cloud support here) The pages you linked to are supposed to serve as client libraries reference only. If you want higher level instructions and examples, always start with the main Google Cloud docs first. On any page that offers instructions, the top of the code windows offers a selection of client library languages/CLI tools/REST API available to do whatever that task is. For cloud storage, start here:

https://cloud.google.com/storage/docs/how-to

Choose a topic, and select "Python" at the top. It should provide instructions and examples using the Python libraries.

Also, we have a repo of demo projects and examples for nearly every GCP product/service, and then some. Great examples to be found here (some might be out of date though):

https://github.com/GoogleCloudPlatform/python-docs-samples

Re: Moving The New York Times Games Platform to Google App Engine

#95

Post by oblio »

obulpathi wrote: post #76
vira28 wrote: post #69

Earlier quoted context omitted.

This thread here came at the right time. Today whole day I attended the DynamoDB training. Honestly, one thing that I understood is its cost based on reads and writes per second. Irrespective of the amount of read data per operation (whether its 1 bytes or 100 bytes), its always charged for 1KB. So, as a work around what they suggested is using a Kinesis, a Lambda and an another service to make the write operation as…

Yep, same problems with many other services: * Kinesis Streams: Writes limited to 1K/sec and 1 MB / shard, reads limited to 2K/shard. Want a different read/write ratio? Nop, not possible. Proposed solution: use more shards. Does not scale automatically. There is another service called Kinesis Streams that does not offer read access to streaming data. * EFS: Cold start problems. If you have small amount of data in EFS…

> * ECS: Two containers can not use same port on same node. Anti pattern to containers.

Could you elaborate for this? I'm not sure I understand, are you saying that 2 containers cannot be mapped to the same host port? Because that would seem normal, you can't bind to a port where there's already something listening. But I guess I must be missing something.

Re: Moving The New York Times Games Platform to Google App Engine

#96

Post by ledgerdev »

funkjunky wrote: post #92
ledgerdev wrote: post #43

Earlier quoted context omitted.

Funny you mention that, I was just experiencing this yesterday in the flex environment. Re-deploy time for a 3 character text change on one page is 8+ minutes for each re-deploy. This is insane. Does anyone know how to speed that up?

(Google cloud support) Unfortunately, this is a pain point that has no easy solution at the moment. It doesn't matter how trivial a change is, it isn't the docker deployment that is taking all that time. As mentioned above, the bottleneck is updating the GC load balancer with the new routing rules, which takes time to propagate throughout the system. This is a high priority issue internally, but updating the load bal…

Great answer, thanks!

Re: Moving The New York Times Games Platform to Google App Engine

#97

Post by _wmd »

vira28 wrote: post #80
ciguy wrote: post #77

As a DevOps consultant I've actually worked with clients migrating stacks to and from GCE/AWS (Yeah, both ways, not the same client). What I've found in aggregate is that GCE is a bit easier to use at first as AWS has a LOT of features and terminology to learn. When it comes down to it though, many GCE services felt really immature, particularly their CloudSQL offering. One client recently moved from GCE to AWS simpl…

As a Database Engineer working for one of the largest e-company in the market, I can clearly see your point. Definitely AWS RDS is very matured when compared to CloudSQL. I think CloudSQL only provides MySQL and Postgres (still in beta?). So GCE needs to build their Database Arsenal soon. Next, your client faced issues with replication in GCE, thats not good to hear, but we do face issues in our AWS RDS MySQL and Aur…

Cloud Postgres is also hilariously hard limited to 100 simultaneous connections (the default). Doesn't matter how much RAM you give it.

My experience with GCP in the past 4 months has led me to revise my "friends don't let friends use App Engine" motto to "friends don't let friends use Google Cloud", there isn't a single service I touched (except maybe Compute Engine) that didn't have half-baked client libraries, documentation, bugs in the server part, or a complete failure by Google to even have their engs use the competitions tooling before inventing their own shitty clone (DNS)

Re: Moving The New York Times Games Platform to Google App Engine

#98

Post by alexeldeib »

oblio wrote: post #95
obulpathi wrote: post #76

Earlier quoted context omitted.

Yep, same problems with many other services: * Kinesis Streams: Writes limited to 1K/sec and 1 MB / shard, reads limited to 2K/shard. Want a different read/write ratio? Nop, not possible. Proposed solution: use more shards. Does not scale automatically. There is another service called Kinesis Streams that does not offer read access to streaming data. * EFS: Cold start problems. If you have small amount of data in EFS…

> * ECS: Two containers can not use same port on same node. Anti pattern to containers. Could you elaborate for this? I'm not sure I understand, are you saying that 2 containers cannot be mapped to the same host port? Because that would seem normal, you can't bind to a port where there's already something listening. But I guess I must be missing something.

I'm not familiar with the details of AWS here, but maybe the OP means mapping two different host ports to the same port on two different containers? That's all I can imagine that would be a container antipattern in the way described.

Re: Moving The New York Times Games Platform to Google App Engine

#100

Post by robotmay »

_wmd wrote: post #97
vira28 wrote: post #80

Earlier quoted context omitted.

As a Database Engineer working for one of the largest e-company in the market, I can clearly see your point. Definitely AWS RDS is very matured when compared to CloudSQL. I think CloudSQL only provides MySQL and Postgres (still in beta?). So GCE needs to build their Database Arsenal soon. Next, your client faced issues with replication in GCE, thats not good to hear, but we do face issues in our AWS RDS MySQL and Aur…

Cloud Postgres is also hilariously hard limited to 100 simultaneous connections (the default). Doesn't matter how much RAM you give it. My experience with GCP in the past 4 months has led me to revise my "friends don't let friends use App Engine" motto to "friends don't let friends use Google Cloud", there isn't a single service I touched (except maybe Compute Engine) that didn't have half-baked client libraries, doc…

I was super keen to switch to GCP (for cost saving etc) but this mirrors a lot of my experiences. Deploys to App Engine took 20 minutes, not 2 minutes, and I have absolutely no faith in their firewall settings actually working. I have no idea what the problem is, but it's basically impossible to boot a Rancher master node on Compute Engine. Even with all ports open. In the end I just bailed on the platform as a whole, and I'm moving to a hybrid approach on smaller providers like Packet.net and Digital Ocean.

And I would have been totally fucked over by that Postgres connection limit when we went into production, I'm glad I dodged that bullet! I hadn't bumped into that when playing with dev environments, and I haven't seen that limit mentioned anywhere.

Post reply on HN