Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

loggly.com

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#11

Post by zimbatm »

general_failure wrote: post #8
zimbatm wrote: post #4

> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslog caches DNS resolution forever. Or the log forwarded doesn't have a buffer and logs get lost.

How does this work regardless? Isn't route53 a DNS system? How does it know whether a service went up/down? EDIT: To reply to myself http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hea... http://docs.aws.amazon.com/Route53/latest/APIReference/API_C...

Unfortunately inside of AWS there is no other solution than to rely on DNS to update the list of IP that resolve from an address (even ELB does that). Route53 has health-checks that can update the list automatically. Uptime is entirely dependent on the TTL of those records and if the clients respects that TTL.

Complementary to that it's also possible to assign reserved IPs to machines so that at least the set of IP is always the same even if the hosts get rotated. Assigning IPs to hosts is not instantaneous either and depends on an API call. Also IPs are tied to a specific region.

In a real datacenter other options are possible like sharing IP addresses between multiple devices and having lower-latency failover. These aren't perfect either and have different failure scenarios.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#14

Post by korzun »

zimbatm wrote: post #4

> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslog caches DNS resolution forever. Or the log forwarded doesn't have a buffer and logs get lost.

Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation.

That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc.

Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced.

I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balancer infrastructure that can offload requests to their cloud instances.

This is a really bizarre post.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#15

Post by hatred »

korzun wrote: post #14
zimbatm wrote: post #4

> If there is an issue with a collector, Route 53 automatically takes it out of the service; our customers won’t see any impact. Except when for example rsyslog caches DNS resolution forever. Or the log forwarded doesn't have a buffer and logs get lost.

Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation. That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc. Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced. I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balan…

I am more worried about the DNS caching issue.

The 2+ hosts failing should not be much of a problem if you have a separate health checker host which does nothing except gathering heart-beats from all the hosts in your fleet and updating the DNS periodically.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#16

Post by korzun »

hatred wrote: post #15
korzun wrote: post #14

Earlier quoted context omitted.

Yeah I don't get their approach. There is no way this will cause 100% delivery if one server fails within that rotation. That chances of failure go up dramatically if 2+ hosts behind round robin fail, etc. Not to mention once hosts resolve this to an IP they will re-use the route. This approach is not balanced. I don't want to be /that/ guy but if they can't scale with ELB they should invest in a dedicated load balan…

I am more worried about the DNS caching issue. The 2+ hosts failing should not be much of a problem if you have a separate health checker host which does nothing except gathering heart-beats from all the hosts in your fleet and updating the DNS periodically.

That's exactly why it's a problem.

If you have 3 hosts and 2 of them go down, in this setup there is more than 50% chance that cached hosts will be trying to connect to a non-existing server.

Also expecting client to perform a DNS lookup every time there is an outgoing log packet is pretty shitty for performance. You can't guarantee near instant DNS server availability for every client.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#17

Post by former_loggly »

Former Loggly employee here. Loggly is at CTO #3 or 4 in about 3 years. The CEO, marketing guy with black turtle neck, "runs" engineering. It is NOT an engineering company and they are on their way to outsourcing all development to India.

Formally they had all of their EC2 instances configured to run without swap and didn't use EBS such that instances would crash 1-3 times a day and lose all data which would require 1-2 day customer restores of data.

Additionally, this Java shop oversubscribed threads on every Solr box which made them restart each Solr instance every hour. To think any revolutionary engineering ideas come from an former Apple marketing wannabee who puts outsourced Indian engineering in place as yes men is a huge stretch.

Let's be honest, Loggly is in huge trouble and can't hire quality engineering talent and as a result is trying to remarket themselves as an engineering driven company as they outsource to India.

Key question isn't..do you use DNS or Elastic Load Balance...it is...what is your VOLUNTARY RATE OF ATTRITION? Hint, really bad!

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#18

Post by mrucci »

Interesting points. Here is a few things you'll miss choosing Route 53 over ELB:

* HTTPS termination.

* Autoscaling group management. By connecting an ELB to an autoscaling group, the logic of registration and deregistration is fully managed behind the scenes. With route53, you have to implement it yourself.

* Minimum autoscaling group size. If you enable ELB health checks, you can rely on the ELB to maintain a group of instances of constant size.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#19

Post by lfuller »

mbell wrote: post #9

There is a rather large technical divide between 'no logs left behind' and relying on DNS lookup to provide that guarantee.

I was thinking the exact same thing while reading this - it reads like a company that doesn't understand the unique challenges involved with distributed computing.

I'm actually in the middle of deciding between Loggly, Papertrail, and Logentries for centralized log management. I guess that cuts it down to two.

Post reply on HN