21–30 of 37 posts
What I can't understand is -
If your logs are in ELK and metrics in prometheus/grafana - why do you need SSH access? Sounds like thats a good problem to solve
[1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-dat...
Earlier quoted context omitted.
> Why should an instance created by an ASG have a host name? It means you can connect to it by just knowing its instance ID. Adding the IP address everywhere also works. There can be some nice SSH config options though, like using a particular key for everything *.prod.myaws.com
That’s the second part. If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. That’s actually the larger issue.
I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive.
I ran out of disk space. Why?
You can attach an ELB to the ASG. Then DNS name to ELB. That's how AWS expects you to use it.
That doesn't solve the problem of the hostname on the EC2 instance itself being the same across all instances thereby making it harder to see what logs came from what hosts. It doesn't solve the problem of allowing you to look at logs and then quickly SSH'ing to a single machine in the ASG.
Install a log agent on the machine like fluentd. Have it inject the host ip and other contextual meta data in to the logs then forward to your central log system?
When you see the error message in your logs, you get the internal ip and can ssh in.
Persistent internal ip’s/hostnames also means you are not treating hosts as ephemeral. It’s always good in the cloud to get things to a point you can just blow away instances and they auto recreate. It’s even possible with traditional services requiring persistent storage. Put the storage on a seperate volume and have the instance startup scripts discover available volumes and attach as required.
I think AWS not having automatic hostnames for ASG instances is a way to lure you into Lambda's, at least that's how I got hooked.
Why should an instance created by an ASG have a host name? These are cattle not pets. I use Serilog for logging with an EC2 enricher that automatically adds the instance Id and the IP address. Since Serilog does structured logging, I can use either an ElasticSearch or Mongo sink and do complex queries. If I routinely need to log into an instance to troubleshoot, I need to be capturing data and sending it to a central…
For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips.
Also, because we had some had some semi stateful legacy instances that where basically lift and shift to AWS, but I wanted to have them in ASG to keep our environment similar until we could refactor them into real cattle.
Our nodes are in ASGs.
Earlier quoted context omitted.
That’s the second part. If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. That’s actually the larger issue.
> If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive. I ran out of disk space. Why?
That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. (https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc...)
Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem.
AWS isn’t going to run out of disk space any time soon. You could also use a lifecycle policy to delete old logs or move them to a lower cost storage depending on your retention policy.
I’m not saying that I have never had to log on to a VM to troubleshoot, but that’s a sign of the need of better logging.
And if my logging infrastructure isn’t good, how pray tell will I troubleshoot my programs running on Lambda or Fargate?
Earlier quoted context omitted.
Why should an instance created by an ASG have a host name? These are cattle not pets. I use Serilog for logging with an EC2 enricher that automatically adds the instance Id and the IP address. Since Serilog does structured logging, I can use either an ElasticSearch or Mongo sink and do complex queries. If I routinely need to log into an instance to troubleshoot, I need to be capturing data and sending it to a central…
I agree, I wouldn't want it any other way nowadays, but back then I had to migrate a lot of legacy system to AWS under pressure. For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips. Also, because we had some had some semi stateful legacy instances that where…
I do this already but from my configuration management system on my instances, but one thing I don't have that I'd love if Route53 would help support is being able to run route53 to handle in-addr.arpa zones for my IP addresses so I can get reverse IP looking for my VPC networks without having to run my own resolver.
Interesting, which configuration management system do you use?
I do this already but from my configuration management system on my instances, but one thing I don't have that I'd love if Route53 would help support is being able to run route53 to handle in-addr.arpa zones for my IP addresses so I can get reverse IP looking for my VPC networks without having to run my own resolver.
We run our reverse zones on route53. It is just a little bit cumbersome but overall works relatively well with vpc private hosted zones.