Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS mistakes to avoid

cloudonaut.io

Post reply on HN
Print view View original

241–250 of 276 posts

Re: AWS mistakes to avoid

#241

Post by wsh91 »

eva1984 wrote: post #183
ma2rten wrote: post #162

Earlier quoted context omitted.

It's not easy, but also not every difficult. The company I work for has two Dev Ops (only one until a year ago) and runs on AWS serving millions of users. We are using MySQL RDS, but are migrating to self-managed Cassandra. We use RabbitMQ instead of SQS. We use Hive/Presto instead of Redshift.

You sure can adapt the open-sourced solution, though I don't think Cassandra/Presto can match up to what RDS/Redshift can offer. The key point is, running those things yourself, you are likely to lose all the nice failover mechanisms/monitoring/auto-snapshotting stuff that AWS offers. To live up to that, it will require you to not only have extensive understanding of the software you are running, but also a considera…

Cassandra in particular doesn't compete with RDS as much as DynamoDB and, respectfully, setting up failover and monitoring and autosnapshotting isn't actually that difficult. I'm a big fan of outsourcing that stuff--the new system I'm building uses AWS Lambda, for example--but when using a managed solution requires compromising on features or tying yourself into knots to make things work, hosting your own is doable, especially in the age of devops. Automation is your friend!

Re: AWS mistakes to avoid

#242

Post by tptacek »

duggan wrote: post #87
eropple wrote: post #49

Earlier quoted context omitted.

Terraform, as an idea, is brilliant. Mitchell and company isolated a hugely important need and tried to fill it, and I give them all the credit in the world for that. Cross-platform cloud provisioning? Gimme. But I cannot in good conscience not relate what a disastrous experience Terraform has been for me at both jobs and clients. Writing reusable code in Terraform is an exercise in frustration due to the extreme clu…

> Full disclosure: I'm building a much, much better provisioner for multi-provider cloud infrastructure. Neither of the projects I recommend are mine; mine's not done yet. One of the convenient things about software that doesn't exist is that it doesn't have any bugs. Let your software speak for itself when it exists; until then, this seems an undeserved critique of software, and a team, that is solving problems ever…

This is a crazy sentiment. I was just considering checking out Terraform, and I'm really glad to have read the previous commenter's experience.

Re: AWS mistakes to avoid

#243

Post by Gigablah »

tptacek wrote: post #242
duggan wrote: post #87

Earlier quoted context omitted.

> Full disclosure: I'm building a much, much better provisioner for multi-provider cloud infrastructure. Neither of the projects I recommend are mine; mine's not done yet. One of the convenient things about software that doesn't exist is that it doesn't have any bugs. Let your software speak for itself when it exists; until then, this seems an undeserved critique of software, and a team, that is solving problems ever…

This is a crazy sentiment. I was just considering checking out Terraform, and I'm really glad to have read the previous commenter's experience.

I've been using Terraform for over a year, maintaining a standard 3 AZ load balanced production cluster.

HCL has improved dramatically, and now that template strings are a thing, most of my variable interpolation issues are solved. However you still can't specify lists as input variables so you frequently have to resort to joining and splitting strings. It's hackish and worse, changing one value in the list will invalidate all other resources that use the variable.

Race conditions and dependency cycles are still a problem. Particularly with auto scaling groups and launch configurations -- I have to migrate them in two steps (create then destroy) to avoid a conflict. Same with EBS volumes, I ended up scripting my instance to attach the volume by itself, otherwise there's ordering issues when destroying and replacing.

There's also missing features, such as the ability to create elasticache redis clusters and cloudformation resources.

I'm still glad that I went with Terraform though. It takes a good amount of time to get around the limitations and bugs, which can be really frustrating, but when it works, it works beautifully.

Re: AWS mistakes to avoid

#244

Post by dblooman »

meddlepal wrote: post #62
zerotosixty wrote: post #7

You know what I've realized that's really important. More AWS tutorials is really needed. There's numerous of new programmers who want to learn AWS, but can't finish building anything because they get buried in documentation. I find there are a lot of high-level abstracted tutorials, but for the new services, there aren't a lot of detailed tutorials. For instance, an implemented cognito->gateway->lambda->dynamodb is…

The real issue is that AWS isn't designed as a tool for product developers. Product developers get asked to use it but do not usually have a clue about good systems engineering. AWS was designed for ops and systems engineers first and foremost.

That may have been true 5 years ago, but a lot of the newer services require developers to just read the documentation, not have a serious amount of background in the setup of the service they are using. Look at things like ECS and Lambda for compute, SQS for messaging etc.

Re: AWS mistakes to avoid

#245

Post by anthony_barker »

Sir_Cmpwn wrote: post #177
nicobn wrote: post #149

Earlier quoted context omitted.

Since you're dealing in absolutes, let me return the favor; you're being a little silly. Yes, Linode beats AWS when it comes to price. On the other hand, Linode's offer is incredibly basic and simplistic. AWS offers service after service that Linode simply doesn't and realistically cannot. Sure, you can emulate a subset of these services (and a subset of their features) using open-source software but at what price ?…

I'm really not trying to compare Linode to AWS here. I just felt that it was necessary to mention that I work for Linode because there _is_ some overlap. I'm really also not trying to sell anyone on Linode in particular. I'm pointing out that from my own personal experience, AWS is grossly overpriced and worse than the alternatives for a large number of use-cases.

Aww and digital ocean customer here(as well internally hosted).

Main advantages I see is Security - aws has security zones, FW ETC FOR FREE Momentum - constant new features(apache spark, lambda, sqs)

Re: AWS mistakes to avoid

#246

Post by kelnos »

boomzilla wrote: post #236
kelnos wrote: post #198

Earlier quoted context omitted.

Not the grandparent, but I have a lot of experience with DynamoDB as well. I migrated a self-managed, sharded redis store to DDB close to a year ago, and what it really depends on are your read and write patterns, and whether or not you can live with the opacity of what DDB is doing behind the scenes. An example: if you provision X capacity units, you're actually provisioning X/N capacity units per partition. AWS is…

You should put some effort on designing the key schema, especially the hash key. Don't do timestamp, or sequence. I had to do a 'encrypted' key schema on my sequence IDs so sequential records are spread into different partitions. Actually it worked out well as I can exposed these encrypted keys externally too. Agreed that DDB documentation could be better on the best practices. I found this deep dive youtube video ve…

Oh, absolutely. The problem is that your key schema also ties you to what kind of queries you can do efficiently. My primary keys actually are very well-distributed across the hash space, but in my initial implementation, one of the global secondary indexes had a key that caused throttling and absolutely destroyed performance. Dropping that index meant losing the ability to do an entire class of queries. In this case, we were (sorta) able to live with that, but I can imagine many cases where that's a problem.

That's actually another instance of the lack of transparency and trial-and-error: "oh hey, writes are getting throttled... no idea why... let's drop this index and see if it helps".

Re: AWS mistakes to avoid

#247

Post by kelnos »

chucky_z wrote: post #219
kelnos wrote: post #198

Earlier quoted context omitted.

Not the grandparent, but I have a lot of experience with DynamoDB as well. I migrated a self-managed, sharded redis store to DDB close to a year ago, and what it really depends on are your read and write patterns, and whether or not you can live with the opacity of what DDB is doing behind the scenes. An example: if you provision X capacity units, you're actually provisioning X/N capacity units per partition. AWS is…

Was the ElastiCache service around when you migrated Redis to Dynamo? This is another alternative I'm considering.

The problem with ElastiCache -- and why I rejected it as an option -- as that they make you define a 30-minute "maintenance window" where AWS can apply patches and reboot your cache instances. In practice I've heard that this happens rarely, and when it does, the downtime is short, but it can in theory cause longer outages.

And in the case of both the redis and memcached backends, if the maintenance requires restarting redis/memcached or rebooting the instance, you lose all data in the cache (at least up until your last backup). For this particular project, that amount of downtime would easily cause a real outage for customers, and was unacceptable.

Re: AWS mistakes to avoid

#248

Post by kcmastrpc »

tie_ wrote: post #212
zerotosixty wrote: post #7

You know what I've realized that's really important. More AWS tutorials is really needed. There's numerous of new programmers who want to learn AWS, but can't finish building anything because they get buried in documentation. I find there are a lot of high-level abstracted tutorials, but for the new services, there aren't a lot of detailed tutorials. For instance, an implemented cognito->gateway->lambda->dynamodb is…

As much as I agree with your point about being hard, I'd also like to add out that there isn't anything even remotely similar to documentation on how a pure dev person could do this outside of AWS, with the lack of managed cognito, API gateway + lambda, and dynamodb. Making those notions and technologies easy and cheap to access AWS suddenly gave devs the idea that they can roll out complex infrastructure on their ow…

This is probably one of my biggest gripes. Our CTO/PMs are like "you weren't able to containerize and get our app autoscaling in a few days? I don't get it? How long will it take? While you're at it, can you implement kinesis/lambda/microservices for the parts of our app that are taking more than 3000ms?"

Me: "I just said I was interested in DevOps and would like to give it a shot, I don't know how long it'll take, I'm working on it."

Re: AWS mistakes to avoid

#249

Post by subway »

obulpathi wrote: post #232
Jhsto wrote: post #216

Earlier quoted context omitted.

I've found Google Compute Engine decent, but with App Engine I ran into difficulties with vendor lock-in. I had to first modify my code to get the app run on my local computer. After that I never got it deployed because GAE requires me to bind their own TCP context for outbound connections. This was a shame because I would have wanted to have my app on GAE and database on Heroku. Just saying that calling 90% of Googl…

Google Cloud is not just App Engine now. Its pretty much the equivalent to aws in features. Let me also say I have PhD in Cloud and big data (graduated in 2013), worked for Amazon aws, worked for an rackspace for year and half, I was a core contributor to OpenStqck for the products OpenStack Poppy and OpenStack Zaqar. I hope you his puts se more weight behind my statements. If you are interested to know the details a…

Cool story, bro.

Re: AWS mistakes to avoid

#250

Post by a-lex »

obulpathi wrote: post #232
Jhsto wrote: post #216

Earlier quoted context omitted.

I've found Google Compute Engine decent, but with App Engine I ran into difficulties with vendor lock-in. I had to first modify my code to get the app run on my local computer. After that I never got it deployed because GAE requires me to bind their own TCP context for outbound connections. This was a shame because I would have wanted to have my app on GAE and database on Heroku. Just saying that calling 90% of Googl…

Google Cloud is not just App Engine now. Its pretty much the equivalent to aws in features. Let me also say I have PhD in Cloud and big data (graduated in 2013), worked for Amazon aws, worked for an rackspace for year and half, I was a core contributor to OpenStqck for the products OpenStack Poppy and OpenStack Zaqar. I hope you his puts se more weight behind my statements. If you are interested to know the details a…

no.
Post reply on HN