AWS mistakes to avoid • Hacker News Board

Re: AWS mistakes to avoid

#231

Post by larrymcp » Sun, Dec 27, 2015, 6:30 PM UTC

obulpathi wrote: post #205

albertoleal wrote: post #195

Earlier quoted context omitted.

What are some alternatives to AWS? Are there realistic alternatives?

Please take a look into Google Cloud. It's now feature equivalent with AWS ( https://cloud.google.com/docs/google-cloud-platform-for-aws-... ). Google Cloud is better than AWS on all fronts other than Relational Databases. Google Cloud has MySQL only. AWS has MySQL, PostreSQL, Oracle, MSSQl. Rest of the services Google beats AWS by a fair and wide margin. I have done a fair amount of benchmarking Google vs AWS in pas…

Thanks; I hadn't seen that that page from Google and that's a pretty good idea to make that available.

You know what would be a cool idea: if Google developed a method of copying an Amazon Machine Image (AMI) to Google Cloud. That would give us an easy way to try out our same servers at Google without having to rebuild everything, as we do not use containers yet.

Re: AWS mistakes to avoid

#232

Post by obulpathi » Sun, Dec 27, 2015, 6:39 PM UTC

Jhsto wrote: post #216

obulpathi wrote: post #206

Earlier quoted context omitted.

If you look at the speed at which Google Cloud has been adding products for past 6 months, you will be amazed. Google Cloud is now feature complete with AWS and 90% of its products are far better than AWS.

I've found Google Compute Engine decent, but with App Engine I ran into difficulties with vendor lock-in. I had to first modify my code to get the app run on my local computer. After that I never got it deployed because GAE requires me to bind their own TCP context for outbound connections. This was a shame because I would have wanted to have my app on GAE and database on Heroku. Just saying that calling 90% of Googl…

Google Cloud is not just App Engine now. Its pretty much the equivalent to aws in features. Let me also say I have PhD in Cloud and big data (graduated in 2013), worked for Amazon aws, worked for an rackspace for year and half, I was a core contributor to OpenStqck for the products OpenStack Poppy and OpenStack Zaqar. I hope you his puts se more weight behind my statements. If you are interested to know the details and why Google Cloud is better that aws for 90% of products, reach out to me.

Re: AWS mistakes to avoid

#233

Post by obulpathi » Sun, Dec 27, 2015, 6:46 PM UTC

furfish wrote: post #223

obulpathi wrote: post #205

Earlier quoted context omitted.

Please take a look into Google Cloud. It's now feature equivalent with AWS ( https://cloud.google.com/docs/google-cloud-platform-for-aws-... ). Google Cloud is better than AWS on all fronts other than Relational Databases. Google Cloud has MySQL only. AWS has MySQL, PostreSQL, Oracle, MSSQl. Rest of the services Google beats AWS by a fair and wide margin. I have done a fair amount of benchmarking Google vs AWS in pas…

You'll need a better source than a Google marketing page to support claims of "feature parity". An objective comparison also shouldn't start with "Please look into Google cloud".

That is why I am sharing my findings in detail at out local meetup. If interested to know more, just shoot an email to me at Obulpathi at gmail. Also see my another comment above.

Re: AWS mistakes to avoid

#234

Post by lern_too_spel » Sun, Dec 27, 2015, 7:01 PM UTC

Jhsto wrote: post #216

obulpathi wrote: post #206

Earlier quoted context omitted.

If you look at the speed at which Google Cloud has been adding products for past 6 months, you will be amazed. Google Cloud is now feature complete with AWS and 90% of its products are far better than AWS.

I've found Google Compute Engine decent, but with App Engine I ran into difficulties with vendor lock-in. I had to first modify my code to get the app run on my local computer. After that I never got it deployed because GAE requires me to bind their own TCP context for outbound connections. This was a shame because I would have wanted to have my app on GAE and database on Heroku. Just saying that calling 90% of Googl…

App Engine has a lot of faults, but your use case sounds incredibly niche. You're using the multiregion availability of App Engine's HTTP servers but want to use Heroku's single AZ database across relatively flaky, slow internet instead of App Engine's multiregion available database on a managed internal SDN. I'm having a hard time imagining a scenario where that setup would make sense.

Re: AWS mistakes to avoid

#235

Post by JabavuAdams » Sun, Dec 27, 2015, 7:07 PM UTC

devonkim wrote: post #204

JabavuAdams wrote: post #139

Earlier quoted context omitted.

This is a fairly naive way of looking at it. Often it's easier to deal with an external organization than to fix your own organization's behaviour. In a lot of pathologically-managed companies or divisions, there is effectively zero chance of an employee being able changing the pathology.

I have heard of a few poorly managed companies pulling out of ruin by using consultants and third party labor to supplement their collapsing workforce. But I've seen poorly managed companies on occasion fire their executives and stabilize as well. People love to spend 7 figures on consultants and refuse to take their advice in favor of suck-ups that recommend that they just slightly change what they're doing or just…

This sounds about right. This is why I don't believe in the "industry is more efficient than government" dogma. Large organizations of people are generally inefficient unless extraordinary care has gone into the engineering and maintenance of the organization.

Re: AWS mistakes to avoid

#236

Post by boomzilla » Sun, Dec 27, 2015, 9:21 PM UTC

kelnos wrote: post #198

chucky_z wrote: post #137

Earlier quoted context omitted.

What about as a kind of Redis stand-in? I'm curious about the DynamoDB PHP session driver. It sounds like you have a lot of experience with Dynamo. All the use-cases I seem to keep coming up with are more for storing global environment keys outside of the environment itself and using a large number of IAM roles to access individual keys, and as a kind of throwaway 'I need to store this somewhere, but it doesn't reall…

Not the grandparent, but I have a lot of experience with DynamoDB as well. I migrated a self-managed, sharded redis store to DDB close to a year ago, and what it really depends on are your read and write patterns, and whether or not you can live with the opacity of what DDB is doing behind the scenes. An example: if you provision X capacity units, you're actually provisioning X/N capacity units per partition. AWS is…

You should put some effort on designing the key schema, especially the hash key. Don't do timestamp, or sequence. I had to do a 'encrypted' key schema on my sequence IDs so sequential records are spread into different partitions. Actually it worked out well as I can exposed these encrypted keys externally too.

Agreed that DDB documentation could be better on the best practices. I found this deep dive youtube video very helpful: https://www.youtube.com/watch?v=VuKu23oZp9Q

Re: AWS mistakes to avoid

#237

Post by nzoschke » Sun, Dec 27, 2015, 9:32 PM UTC

querulous wrote: post #175

nzoschke wrote: post #88

Earlier quoted context omitted.

We are working to normalize the AWS scenario for all apps. Every Convox cluster is an autoscale group managed cluster of ECS instances. Every Convox app gets its own Kinesis stream for logs, ELB for load balancing, and S3 buckets for settings, build artifacts and encrypted environment. And the app processes are run via ECS. So I'm confident we could handle the 7 apps in a single cluster, scale the cluster instance si…

i appreciate you taking the time to reply, thanks. some additional questions though: 1. are security groups opaque within convox or are they exposed to developers? 2. when you say monitoring tools are built in and you have tools for logging does this lock me in to the convox log pipeline and monitoring? what i want to use sensu on my instances? do i have to add sensu to every container? if you run, for instance, five…

1. Devs don't have to worry about security groups with `convox install && convox deploy && convox ssh`. But security groups are created with CloudFormation and you can use your AWS keys to introspect and change them.

2. Currently every app gets a Kinesis stream and we tail all Docker logs and put them into Kinesis. Then `convox logs` can stream logs from Kinesis, and `convox services add papertrail` adds a Lambda / Kinesis event source mapping to emit the stream as syslog to Papertrail.

I'm pretty happy with this setup and think it represents a good default infrastructure that is still extensible.

Would Kinesis -> Lambda -> Sensu make sense too? It's a pretty new pattern but this seems a lot saner to me than per-container log agents, or even bothering with custom logging drivers.

That said, one user has been using logstash by bringing a custom AMI with his logstash agent and creds baked in.

3. It's one VPC per rack, but I could see modifying that. We've already started to parameterize some VPC settings like the CIDR block to help integrating with your existing VPC usage.

https://github.com/convox/rack/blob/master/api/dist/kernel.j...

Re: AWS mistakes to avoid

#238

Post by randomflavor » Sun, Dec 27, 2015, 9:35 PM UTC

jasode wrote: post #197

albertoleal wrote: post #195

Earlier quoted context omitted.

What are some alternatives to AWS? Are there realistic alternatives?

The "alternatives" depend on your situation. Picture a continuum between brain-dead simple websites and business-critical complex websites: simple: static website, WordPress blog moderate: small business CMS, etc complex: Netflix, AirBNB If you're running a simple WordPress blog, the AWS prices are absurdly overkill. For this use-case, there are a zillion alternatives. Linode, Digital Ocean, Rackspace bare metal, etc…

Digitalocean is a great alternative for simple things.

I use it for dev / early projects and as things get complex or need more redundancy I make the production spend on AWS.

Re: AWS mistakes to avoid

#239

Post by nzoschke » Sun, Dec 27, 2015, 9:39 PM UTC

rybosome wrote: post #161

nzoschke wrote: post #60

Earlier quoted context omitted.

Convox member here. Thanks for the shout out. This is definitely a goal of Convox: to remove as much AWS complexity as possible. Our approach matches this guide to a tee. We are using CloudFormation to set up a private app cluster, as well as to create and update (deploy) apps. We are also using ASGs. The instance utilization point is spot on too. The fist thing convox does to make this easy is a single command to re…

Any plans to support the other public cloud providers?

Short term, no.

The plan is to get the Convox API locked in while mastering advanced AWS like VPC, ECS, ELB, Kinesis and Lambda behind the scenes.

Long term, yes, and in tandem with when the other cloud providers leveling up. For example Google Cloud Logging (for continer logs on GCE) is still in beta.

Re: AWS mistakes to avoid

#240

Post by wsh91 » Sun, Dec 27, 2015, 10:49 PM UTC

greenleafjacob wrote: post #38

alttab wrote: post #2

Also, dont use Dynamo unless you have a really good reason. "It scales better than MySQL" is not a good reason. Have fun migrating data and re-indexing constantly!

Is local secondary indices and lower operational costs than Cassandra a good reason(s)?

Operational costs are in the eye of the beholder. We evaluated DynamoDB for a new use case but we're going with Cassandra because the storage costs alone more than made up for ops overhead. At least IMO, Cassandra's support for multicolumn range queries (via clustering columns) and the cheaper storage you can use obviate local secondary indices. (Cassandra has secondary indices as well but it seems like most folks prefer further denormalization--at least, that's what we're doing.)