Former Loggly employee here. Loggly is at CTO #3 or 4 in about 3 years. The CEO, marketing guy with black turtle neck, "runs" engineering. It is NOT an engineering company and they are on their way to outsourcing all development to India. Formally they had all of their EC2 instances configured to run without swap and didn't use EBS such that instances would crash 1-3 times a day and lose all data which would require…
Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
31–40 of 50 posts
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#32> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslog caches DNS resolution forever. Or the log forwarded doesn't have a buffer and logs get lost.
Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation. That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc. Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced. I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balan…
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#33Interesting points. Here is a few things you'll miss choosing Route 53 over ELB: * HTTPS termination. * Autoscaling group management. By connecting an ELB to an autoscaling group, the logic of registration and deregistration is fully managed behind the scenes. With route53, you have to implement it yourself. * Minimum autoscaling group size. If you enable ELB health checks, you can rely on the ELB to maintain a group…
ELB scales horrible and can not scale to even tens of thousands of connections per second, let alone handling spikes of 100k/sec simultaneous connections. Even if you get AWS to prewarm to ELB at a higher peak rate, if you spike over those limits you will drop new incoming connections. HTTPS termination is trivial compared to a requirement to be able to actually handle hundreds of thousands to millions of simultaneou…
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#34Earlier quoted context omitted.
I'm sure it's because rsyslog supports it (many mentions in the article to staying compliant with rsyslog).
Thats what I don't get, how is it responsible to stay compliant when that means insecure?
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#35Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#36Former Loggly employee here. Loggly is at CTO #3 or 4 in about 3 years. The CEO, marketing guy with black turtle neck, "runs" engineering. It is NOT an engineering company and they are on their way to outsourcing all development to India. Formally they had all of their EC2 instances configured to run without swap and didn't use EBS such that instances would crash 1-3 times a day and lose all data which would require…
I interviewed there for a devops/sys admin role and ran far away after that process when I learned about what is going on and problems the ops group has to solve. Then reading this fluff piece made me glad I never even thought about working there after that phone interview. Whoever claims a DNS round robin is a good way to handle fail over doesn't really know what they are talking about. I have dug into the how somet…
This is unrelated to the loggly bit, it would just be interesting to know for a non-ops-guy.
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#37In their docs, Loggly only gives out one API endpoint: logs-01.loggly.com.
It is referenced as the endpoint for HTTP, HTTPS, syslog and syslog TLS. These seem to be the only methods available to send log data to them.
There is the obvious problem that a DNS record with a 60s TTL cannot possibly receive every single packet sent to it in the event of a server failure. Even if the returned IP address is an elastic IP, it takes a substantial amount of time to move to another instance in AWS.
I don't know why you would use the same service hostname for all of these endpoints. Separate names for each endpoint, even if they all pointed to the same pool of hosts, would at least give some flexibility in the future when they have enough traffic to get desperate about capacity. I would also think they might want to segregate native syslog from HTTP traffic, since I presume it uses different processes on the backend.
It's also curious that they chose to return only one A record. DNS RR is a poor substitute for real load balancing, but it's better than nothing. With multiple A records, there is at least a chance that some of their traffic will go to other servers -- rather than all of it potentially going to one as it is now.
While they made no claims about using Route 53 for its geo DNS capabilities, I still found it amusing that I was sent to a US East IP from California. Not that it's super critical that my log lines get delivered quickly, but it is ideal to shorten the path of an insecure and unreliable transport in order to improve durability. Although I would never ship syslog out to some host on the Internet, a host 16 hops away is even more ludicrous.
I think their article says a lot more about how poorly ELBs function when you exceed the low traffic threshold it is seemingly designed for than about how well Route 53 works (and it is a decent static DNS service). The inability to robustly direct incoming traffic is the achilles heel of AWS.
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#38Serious question: Are people upvoting this to poke fun like some kind of daily wtf? A logging platform that lists 1 of their 2 major requirements as "To not drop any data, ever" is using round robin DNS for fault tolerance? I can't see too many people on HN upvoting this for being insightful or impressive. Edit: I just can't help myself. How are you going to send syslog when any server fails and not "drop any data, e…
I upvoted it in the hopes that someone would provide the missing piece. Like "oh, we forgot to mention that the DNS is pointing to our own haproxy servers that all have redundant power/network/whatever) or something.
EC2 instances running haproxy would mitigate a number of the problems they discussed with using ELBs but the inability to use VIPs (with vrrp or ucarp) in AWS means that a failure will always boil down to the same pattern: a key front end instance dies, client traffic keeps being directed to it for 5 minutes (at best), and that's life.
Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#39Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing
#40What are some alternatives to Loggly? I really like being able to aggregate my logs with minimal setup (and cost). I'm logging with Logback (Java), and there is a convenient extension that forwards log statements to Loggly.