Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

loggly.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#41

Post by troydavis »

ejain wrote: post #35

What are some alternatives to Loggly? I really like being able to aggregate my logs with minimal setup (and cost). I'm logging with Logback (Java), and there is a convenient extension that forwards log statements to Loggly.

We just finished writing a syslog4j-derived Logback appender with support for UDP, TCP with TLS encryption, and cleartext TCP:

Background and setup: http://help.papertrailapp.com/kb/configuration/java-logback-...

GitHub repo: https://github.com/papertrail/logback-syslog4j

Papertrail also works with the standard Logback SyslogAppender.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#42

Post by gk1 »

ejain wrote: post #35

What are some alternatives to Loggly? I really like being able to aggregate my logs with minimal setup (and cost). I'm logging with Logback (Java), and there is a convenient extension that forwards log statements to Loggly.

Have a look at Scalyr: https://www.scalyr.com

The setup is straightforward [0], and you'll find that the plans are very competitive.

[0] https://www.scalyr.com/help/install-agent

(Full disclosure: I work with them.)

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#43

Post by yid »

boulos wrote: post #31
former_loggly wrote: post #17

Former Loggly employee here. Loggly is at CTO #3 or 4 in about 3 years. The CEO, marketing guy with black turtle neck, "runs" engineering. It is NOT an engineering company and they are on their way to outsourcing all development to India. Formally they had all of their EC2 instances configured to run without swap and didn't use EBS such that instances would crash 1-3 times a day and lose all data which would require…

I'm disappointed this is the top rated comment. Your repeated bigotry towards "outsourced Indian engineering" combined with using a throwaway detract from what appears to be potentially useful information (Loggly CTO changes, running without EBS and without swap, etc.). Can someone else corroborate the factual bits without the xenophobia?

Outsourced teams anywhere generally don't get much ownership of the product. What results is a bare minimum meet-the-specs effort, which is understandable but generally results in a crappy product.

This is a reflection of the social and communication issues in typical outsourcing setups, not a reflection of the talent of the outsourced team.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#44

Post by toomuchtodo »

hijinks wrote: post #21
former_loggly wrote: post #17

Former Loggly employee here. Loggly is at CTO #3 or 4 in about 3 years. The CEO, marketing guy with black turtle neck, "runs" engineering. It is NOT an engineering company and they are on their way to outsourcing all development to India. Formally they had all of their EC2 instances configured to run without swap and didn't use EBS such that instances would crash 1-3 times a day and lose all data which would require…

I interviewed there for a devops/sys admin role and ran far away after that process when I learned about what is going on and problems the ops group has to solve. Then reading this fluff piece made me glad I never even thought about working there after that phone interview. Whoever claims a DNS round robin is a good way to handle fail over doesn't really know what they are talking about. I have dug into the how somet…

DevOps here. What was the red flag? (Curious for the next time I'm interviewing)

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#45

Post by toomuchtodo »

skuhn wrote: post #38
latch wrote: post #29

Earlier quoted context omitted.

I upvoted it in the hopes that someone would provide the missing piece. Like "oh, we forgot to mention that the DNS is pointing to our own haproxy servers that all have redundant power/network/whatever) or something.

Loggly seems to be all about running In The Cloud, so that seems unlikely. EC2 instances running haproxy would mitigate a number of the problems they discussed with using ELBs but the inability to use VIPs (with vrrp or ucarp) in AWS means that a failure will always boil down to the same pattern: a key front end instance dies, client traffic keeps being directed to it for 5 minutes (at best), and that's life.

The only way for them to fix this would be to have their own client that would attempt retries while caching unacked logs locally, but they market not needing a client (only syslog) as the benefit.

TL;DR Loggly can't promise no data lose in its current incarnation.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#46

Post by ejain »

troydavis wrote: post #41
ejain wrote: post #35

What are some alternatives to Loggly? I really like being able to aggregate my logs with minimal setup (and cost). I'm logging with Logback (Java), and there is a convenient extension that forwards log statements to Loggly.

We just finished writing a syslog4j-derived Logback appender with support for UDP, TCP with TLS encryption, and cleartext TCP: Background and setup: http://help.papertrailapp.com/kb/configuration/java-logback-... GitHub repo: https://github.com/papertrail/logback-syslog4j Papertrail also works with the standard Logback SyslogAppender.

Does the syslog appender handle large, multi-line log messages (i.e. messages containing stack traces)?

I recall having some trouble with that when using syslog with Loggly, before switching over to the json appender.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#47

Post by tkfx »

ejain wrote: post #35

What are some alternatives to Loggly? I really like being able to aggregate my logs with minimal setup (and cost). I'm logging with Logback (Java), and there is a convenient extension that forwards log statements to Loggly.

Hi ejain, check out this post by Takipi, the 7 log management tools you need to know: http://www.takipiblog.com/the-7-log-management-tools-you-nee...

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#48

Post by troydavis »

ejain wrote: post #46
troydavis wrote: post #41

Earlier quoted context omitted.

We just finished writing a syslog4j-derived Logback appender with support for UDP, TCP with TLS encryption, and cleartext TCP: Background and setup: http://help.papertrailapp.com/kb/configuration/java-logback-... GitHub repo: https://github.com/papertrail/logback-syslog4j Papertrail also works with the standard Logback SyslogAppender.

Does the syslog appender handle large, multi-line log messages (i.e. messages containing stack traces)? I recall having some trouble with that when using syslog with Loggly, before switching over to the json appender.

Short answer: it depends.

Long answer: logback and both appenders can accept pattern formats to adjust how they're formatted. How useful the end result is depends a lot on the receiver, though, and more than that, there's no one implementation that's great for everyone -- that is, there's no right way to "handle large, multi-line log messages," only attempts at making them more useful.

An easy example is searching. Some people want to see the entire message, others want only the matching portion of a stack trace, others want some combination, and others - probably most people - just want something that's useful, however the actual UX works.

In Papertrail's case, our sender-specific context links (think grep -A/-B/-C) were designed for navigating multiline output from a single sender: https://papertrailapp.com/tour/viewer/context. It's basically pivoting from a single entry in a stack trace to the entire stack trace.

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#49

Post by ejain »

troydavis wrote: post #48
ejain wrote: post #46

Earlier quoted context omitted.

Does the syslog appender handle large, multi-line log messages (i.e. messages containing stack traces)? I recall having some trouble with that when using syslog with Loggly, before switching over to the json appender.

Short answer: it depends. Long answer: logback and both appenders can accept pattern formats to adjust how they're formatted. How useful the end result is depends a lot on the receiver, though, and more than that, there's no one implementation that's great for everyone -- that is, there's no right way to "handle large, multi-line log messages," only attempts at making them more useful. An easy example is searching. S…

My main concern isn't formatting or searching, but truncation, see http://stackoverflow.com/questions/2011986/does-syslog-reall....

Re: Why Loggly Chose AWS Route 53 Over Elastic Load Balancing

#50

Post by dbarlett »

spamizbad wrote: post #26
mrucci wrote: post #18

Interesting points. Here is a few things you'll miss choosing Route 53 over ELB: * HTTPS termination. * Autoscaling group management. By connecting an ELB to an autoscaling group, the logic of registration and deregistration is fully managed behind the scenes. With route53, you have to implement it yourself. * Minimum autoscaling group size. If you enable ELB health checks, you can rely on the ELB to maintain a group…

ELB's HTTPS termination is mediocre and, last I checked, doesn't offer the best ciphers. A year ago It was impossible to get an A+ on ssltest https://www.ssllabs.com/ssltest/ using ELB to terminate SSL. Not to mention it still needlessly includes a ton of dangerously insecure ciphers just begging to be misclicked.

The current default, ELBSecurity Policy-2014-01 [1] enables ECDSA/PFS and is close to the Mozilla TLS recommendations [2]. Getting an A+ on the Qualys test requires the HSTS header [3], which isn't an ELB issue.

[1] http://docs.aws.amazon.com/ElasticLoadBalancing/latest/Devel...

[2] https://wiki.mozilla.org/Security/Server_Side_TLS#Amazon_Web...

[3] http://mir.aculo.us/2014/04/04/how-to-get-an-a-on-the-qualsy...

Post reply on HN