Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamic Route53 records for AWS auto scaling groups with Terraform

underthehood.meltwater.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#31

Post by paulddraper »

scarface74 wrote: post #27
paulddraper wrote: post #22

Earlier quoted context omitted.

> If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive. I ran out of disk space. Why?

If you’re logging to a local disk on ephemeral VMs, that doesn’t make the situation any better. That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. ( https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc... ) Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem…

I never said your disk was full with logs.

> how pray tell will I troubleshoot my programs running on Lambda or Fargate?

That is indeed a big problem running on Lambda and Fargate.

In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#32

Post by scarface74 »

paulddraper wrote: post #31
scarface74 wrote: post #27

Earlier quoted context omitted.

If you’re logging to a local disk on ephemeral VMs, that doesn’t make the situation any better. That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. ( https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc... ) Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem…

I never said your disk was full with logs. > how pray tell will I troubleshoot my programs running on Lambda or Fargate? That is indeed a big problem running on Lambda and Fargate. In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.

It’s not a problem at all with lambda or Fargate. Logging can be as simple as printing to the console and they go to CloudWatch.

It’s the same concept. If you’re troubleshooting at any point involves needing to log in to an EC2 instance, you might as well have a few bespoke servers called “Web01” and “Web02”. You’re just using ASG to create pets at scale. We run an ASG in production that scales from 2 to 30 instances based on the number of messages in a queue, lambdas running all of the time, some a Fargate tasks etc. it would be a nightmare to troubleshoot all of those processes without centralized, queryable logs.

In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.

And that experience is representative of the entire AWS ecosystem?

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#33

Post by aequitas »

scarface74 wrote: post #28
aequitas wrote: post #24

Earlier quoted context omitted.

I agree, I wouldn't want it any other way nowadays, but back then I had to migrate a lot of legacy system to AWS under pressure. For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips. Also, because we had some had some semi stateful legacy instances that where…

Just out of curiosity, why not just put the ASG behind a load balancer?

I don't remember exactly. We did use elb's for all other services. So it was either cost or it had to do with MX record restrictions in that you're not allowed to use CNAMEs in MX records.

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#34

Post by jimsheldon »

cxmcc wrote: post #16

You can attach an ELB to the ASG. Then DNS name to ELB. That's how AWS expects you to use it.

Post author here.

We absolutely put an ELB/ALB in front of these ASGs as well. The post mentions a few use cases where unique hostnames with internal Route53 records are helpful for us.

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#35

Post by jimsheldon »

fatninja wrote: post #21

Can't this be solved by using IP addresses for hostnames? This can be a part of bootstrap script(which ASG/Launch Configuration already supports via UserData[1]) What I can't understand is - If your logs are in ELK and metrics in prometheus/grafana - why do you need SSH access ? Sounds like thats a good problem to solve [1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-dat...

Post author here.

SSH access is a last resort, but it can be necessary in certain cases. For example, if our log forwarding breaks. SSH is also just one example, it can also be helpful to curl endpoints on the host directly without hitting the ELB/ALB.

The post actually provides the user_data script we use.

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#36

Post by jimsheldon »

OJFord wrote: post #25

Why are ASGs 'outdated' in a world with Kubernetes? Our nodes are in ASGs.

Post author here.

I brought up that point since I think most developers prefer the user experience of Lambda/Kubernetes where they don't have to manage individual instances in Auto Scaling Groups. They certainly are not 'outdated' for our use cases, and especially not for those responsible for running the underlying infrastructure (when running Kubernetes nodes).

Re: Dynamic Route53 records for AWS auto scaling groups with Terraform

#37

Post by jimsheldon »

scarface74 wrote: post #10
paulddraper wrote: post #8

Earlier quoted context omitted.

> Why should an instance created by an ASG have a host name? It means you can connect to it by just knowing its instance ID. Adding the IP address everywhere also works. There can be some nice SSH config options though, like using a particular key for everything *.prod.myaws.com

That’s the second part. If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. That’s actually the larger issue.

Post author here.

SSH access is absolutely a last resort, but can be necessary in certain cases (like when Filebeat breaks...). Turning SSH off completely (i.e. "No SSH") is certainly better for security and something we may pursue.

I mentioned in another comment here that SSH is just one example, we can also easily hit endpoints with curl via hostname.

Also mentioned in the post are other tools (like Grafana dashboards) have an expectation of unique hostnames.

Post reply on HN