Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

packetzoom.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#31

Post by Karunamon »

lobster_johnson wrote: post #30
makapuf wrote: post #10

What about simple rsyslog ? I stumble on this kind of programs (others have mentioned heka, fluentd, logstash), but the general speed, simplicity, versatility -the feature range is actually quite big from ES output to unix pipes to simple filters - and ubiquity of rsyslog make it suited for many of these tasks. I am missing something ?

We would switch away from Rsyslog in a heartbeat if someone could come up with a better syslog-compatible forwarder. We have it set up to write logs locally (with a limited rotation) as well as forward them via TLS to a central Rsyslog server that collects the log in a single tree with a much longer retention time. (We don't use any of the non-file outputs, but we do sync to S3 for archival.) It has major issues. For…

Not sure what you mean by forwarding support - my entire environment is configured with syslog-ng forwarding to various places based on various rules.

Heck of a lot faster than rsyslog. The one thing I've not been able to do is get rsyslog forwarding to syslog-ng. Something happens to the message format between systems that leads to hilariously incorrect filenames on the collector systems.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#32

Post by whost49 »

shlant wrote: post #26

So as someone who is just about to implement Fluentd, what is the status of using LogZoom with docker? Currently, with Fluentd all I have to do is set the log-driver and tags in DOCKER_OPTS, point fluentd to ES, and I have all my container logs. Does LogZoom work this seemlessly with docker? I know that at the very least I will need https://github.com/docker/docker/issues/20363 in order to implement any LogZoom plugi…

For your use case, I think Fluentd may work fine. LogZoom currently deals with structured JSON log data received from hundreds of hosts around the world. It could be modified to handle arbitrary logs (and wrap a structure around it) and integrate with Docker, but that was not the goal here.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#33

Post by lobster_johnson »

Karunamon wrote: post #31
lobster_johnson wrote: post #30

Earlier quoted context omitted.

We would switch away from Rsyslog in a heartbeat if someone could come up with a better syslog-compatible forwarder. We have it set up to write logs locally (with a limited rotation) as well as forward them via TLS to a central Rsyslog server that collects the log in a single tree with a much longer retention time. (We don't use any of the non-file outputs, but we do sync to S3 for archival.) It has major issues. For…

Not sure what you mean by forwarding support - my entire environment is configured with syslog-ng forwarding to various places based on various rules. Heck of a lot faster than rsyslog. The one thing I've not been able to do is get rsyslog forwarding to syslog-ng. Something happens to the message format between systems that leads to hilariously incorrect filenames on the collector systems.

By forwarding I mean reliable, disk-buffered forwarding. This only exists in the commercial "Premium Edition" of syslog-ng.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#34

Post by b0ti »

lobster_johnson wrote: post #33
Karunamon wrote: post #31

Earlier quoted context omitted.

Not sure what you mean by forwarding support - my entire environment is configured with syslog-ng forwarding to various places based on various rules. Heck of a lot faster than rsyslog. The one thing I've not been able to do is get rsyslog forwarding to syslog-ng. Something happens to the message format between systems that leads to hilariously incorrect filenames on the collector systems.

By forwarding I mean reliable, disk-buffered forwarding. This only exists in the commercial "Premium Edition" of syslog-ng.

There is disk based buffering in NXLog CE. You might want to check it out with respect to the other woes you have with rsyslog.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#35

Post by otterley »

lobster_johnson wrote: post #33
Karunamon wrote: post #31

Earlier quoted context omitted.

Not sure what you mean by forwarding support - my entire environment is configured with syslog-ng forwarding to various places based on various rules. Heck of a lot faster than rsyslog. The one thing I've not been able to do is get rsyslog forwarding to syslog-ng. Something happens to the message format between systems that leads to hilariously incorrect filenames on the collector systems.

By forwarding I mean reliable, disk-buffered forwarding. This only exists in the commercial "Premium Edition" of syslog-ng.

... which I highly recommend until you outgrow it and graduate to Kafka.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#36

Post by otterley »

I'm a bit concerned that you're relying on RedisMQ for buffering. Redis is an in-memory store with optional persistence, but having persistence doesn't make it a log-structured system like Kafka. You still have to make sure you don't run out of memory. This greatly limits its ability to buffer messages.

It would have been much better IMHO to utilize an on-disk buffer instead, like syslog-ng PE does.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#37

Post by lobster_johnson »

b0ti wrote: post #34
lobster_johnson wrote: post #33

Earlier quoted context omitted.

By forwarding I mean reliable, disk-buffered forwarding. This only exists in the commercial "Premium Edition" of syslog-ng.

There is disk based buffering in NXLog CE. You might want to check it out with respect to the other woes you have with rsyslog.

Never heard of that one, thanks.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#38

Post by lobster_johnson »

otterley wrote: post #35
lobster_johnson wrote: post #33

Earlier quoted context omitted.

By forwarding I mean reliable, disk-buffered forwarding. This only exists in the commercial "Premium Edition" of syslog-ng.

... which I highly recommend until you outgrow it and graduate to Kafka.

On the other hand, it's pricy and last I checked, licensing is based on the number of machines (ridiculous in a cloud environment).

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#39

Post by dozzie »

stephenr wrote: post #12
makapuf wrote: post #10

What about simple rsyslog ? I stumble on this kind of programs (others have mentioned heka, fluentd, logstash), but the general speed, simplicity, versatility -the feature range is actually quite big from ES output to unix pipes to simple filters - and ubiquity of rsyslog make it suited for many of these tasks. I am missing something ?

It isn't new and shiny and its available from distro packages, so it's not worth the attention of the cool kids. If you don't need to curl|sh from a .io (or .sh) domain to install it, it's not worth using apparently.

Just for information, I generally agree with what you said here, except in this particular case rsyslog is not a generic data bus, and Fluentd and logstash are, so they're useful on their own merit. They're just often used as merely log transports, which overlaps with rsyslog.

Re: LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

#40

Post by ktamura »

chetanahuja wrote: post #4
latch wrote: post #2

Operationally speaking the single most important thing you should be doing is collecting application and system logs and having them easily accessible and usable (and check your backups every now and again). I say this with respect to the value you gain in comparison to the relatively small costs. You're being your own worst enemy if you aren't staying on top of error logs. The OSS solutions are mature and simple to…

PacketZoom founder here. Glad you liked the project. Could not agree more with the importance of tracking logs (and metrics... but that's a topic for another post). To respond to your point about absence of Grok like facility, avoiding the need to unmarshal and remarshal the data while passing through LogZoom was an explicit design requirement. The blogpost refers to our pain with Logstash/Fluentd etc. We were in a s…

>To respond to your point about absence of Grok like facility, avoiding the need to unmarshal and remarshal the data while passing through LogZoom was an explicit design requirement. The blogpost refers to our pain with Logstash/Fluentd etc.

I think there are two different (CPU) performance problems conflated into one:

(1) The cost of parsing logs with something like Grok and Regexp

(2) The cost of marshaling and unmarshaling data

While both do cost CPU time, based on my experience having talked to literally hundreds of Fluentd users (I'm a maintainer and was a core support member for awhile), the cost of (1) dwarfs the cost of (2). (2) is pretty cheap if you use efficient serializers like MessagePack. As for (1), both Logstash and Fluentd support an option to perform zero parsing (In Fluentd, it's "format none"). By using these options, you can bring down CPU time significantly.

All of this being said, it looks like LogZoom isn't a true competitor to Fluentd or Logstash or Heka. It made different performance/functionality trade-offs and by doing less, it saves more CPU time: If you forgo the option of parsing logs at source (and in Logstash and Fluentd's defense, they do a whole lot more), you obviously can save resources. On the flip side, you need to post-process your logs to make them useful, and some other servers downstream will pay for CPU (You might not care about this because your logs have been thrown over the fense and now it's data engineers's job =p)

Post reply on HN