Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Another Redis case: Centralized logging

sunilarora.org

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Another Redis case: Centralized logging

#31

Post by tptacek »

moe wrote: post #30
tptacek wrote: post #12

Earlier quoted context omitted.

Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a gr…

Yes, syslog is a pile of shit. It's a relic. And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running. However... Freeform text is a terrible way to track system events. Nothing stops you from logging structured text. Periodically rotated flat files…

1.

It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on.

2.

Most syslog daemons will store anywhere. Sure. The last syslog daemon I actually read had stack overflows in it, so it's been awhile for me. But if you've got syslog writing to a real store, what value is syslog adding? The standardized network protocol? Redis has a trivial network protocol too.

3.

The issue with syslog facilities isn't that the "UUCP" facility harms anything directly; it's that one the few bits of out-of-band metadata syslog truly offers forces applications to decide whether they're "kern" or "daemon" or "local2". Whose application actually breaks down cleanly into syslog facilities?

4.

I dispute that syslog is "good enough if you're not Facebook". I think you do too: you probably aren't doing all your metrics stuff with syslog logs; you probably aren't even getting your web stats though syslog. You may have even delegated your web logs to client side Javascript, because that is how bad text logs stored in flat files are to work with.

Re: Another Redis case: Centralized logging

#32

Post by moe »

tptacek wrote: post #14
andrewvc wrote: post #4

My only question: how is this better than syslog?

It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just t…

I think you're missing the forest for the trees.

What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.

Re: Another Redis case: Centralized logging

#33

Post by cdavid »

tptacek wrote: post #14
andrewvc wrote: post #4

My only question: how is this better than syslog?

It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just t…

I don't really understand your list in view of the article. How is this solution more queryable than syslog: they record events into redis without any schema related to it (just a string), so I fail to see the improvement there. They put it back to a log file anyway.

It is not less or more centralized than syslog configured with centralization (which is trivial to set up).

How is this more common than syslog across platforms (unless you include windows in "across platforms ?").

It is not simpler than syslog either, since writing to syslog is just a matter of using the right python logging backend.

Analysing tons of data from syslog is a pain, but I don't see how any solution will not require at some point in the stack to enforce a format/structure in your log. How is this fundamentally different than post-processing http log ?

Re: Another Redis case: Centralized logging

#34

Post by tptacek »

moe wrote: post #32
tptacek wrote: post #14

Earlier quoted context omitted.

It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just t…

I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.

Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.

Re: Another Redis case: Centralized logging

#35

Post by tptacek »

cdavid wrote: post #33
tptacek wrote: post #14

Earlier quoted context omitted.

It's queryable. It's trivially capped and rolled. It's centralized. It provides a common logging interface across platforms. It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long). I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just t…

I don't really understand your list in view of the article. How is this solution more queryable than syslog: they record events into redis without any schema related to it (just a string), so I fail to see the improvement there. They put it back to a log file anyway. It is not less or more centralized than syslog configured with centralization (which is trivial to set up). How is this more common than syslog across p…

He's LPUSH'ing logs into a list key. Just by doing that one thing, he now has evented logging; he can subscribe to his raw logs with BRPOPLPUSH and without changing anything clientside start indexing them in better ways, or applying different policies to different logs.

And Redis is easier to understand than syslog. We're pretending that there is zero friction to understanding syslog, as if any competent Unix person should automatically grok it. But syslog is a janky old rube goldberg machine and understanding it well pays off solely in the form of understanding a janky old rube goldberg machine that nobody will be using in 10 years.

Re: Another Redis case: Centralized logging

#36

Post by moe »

tptacek wrote: post #31
moe wrote: post #30

Earlier quoted context omitted.

Yes, syslog is a pile of shit. It's a relic. And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running. However... Freeform text is a terrible way to track system events. Nothing stops you from logging structured text. Periodically rotated flat files…

1. It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on…

what value is syslog adding? The standardized network protocol?

Exactly. Every piece of software under the sun either logs to syslog by default or has a config-switch/patch to make it so.

Yes, syslog the protocol is pretty nasty. Yes, syslog daemons are pretty nasty. But they get the job done and you are free to put anything (including redis) at the end of the pipe.

Coping with syslog is by far easier than trying to make all "legacy software" including the kernel speak something else entirely.

Re: Another Redis case: Centralized logging

#37

Post by tptacek »

moe wrote: post #36
tptacek wrote: post #31

Earlier quoted context omitted.

1. It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on…

what value is syslog adding? The standardized network protocol? Exactly. Every piece of software under the sun either logs to syslog by default or has a config-switch/patch to make it so. Yes, syslog the protocol is pretty nasty. Yes, syslog daemons are pretty nasty. But they get the job done and you are free to put anything (including redis) at the end of the pipe. Coping with syslog is by far easier than trying to…

But who's suggesting that? Nobody is saying "let's have the kernel log to Redis". By all means, hot potato the kernel stuff and the wrapper stuff and your authlog and whatever through syslog before it gets dumped into Redis.

But if you control it, why bother with syslog? Syslog is a piece of junk. Don't bother. Any ad hoc scheme you come up with that uses a real backend store will be better than syslog.

Re: Another Redis case: Centralized logging

#38

Post by moe »

tptacek wrote: post #34
moe wrote: post #32

Earlier quoted context omitted.

I think you're missing the forest for the trees. What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.

Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.

Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel?

Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.

Re: Another Redis case: Centralized logging

#39

Post by tptacek »

moe wrote: post #38
tptacek wrote: post #34

Earlier quoted context omitted.

Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.

Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself , speak this fancy new protocol.

If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.

Re: Another Redis case: Centralized logging

#40

Post by moe »

tptacek wrote: post #39
moe wrote: post #38

Earlier quoted context omitted.

Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel? Because that's exactly what I'd have to do until other daemons like, say, redis itself , speak this fancy new protocol.

If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.

No special new logging protocol is required.

So, what do I put into the redis configuration-file to make it log to itself?

Post reply on HN