Live data from Hacker News

Linux syslog on the way out?

itworld.com

31–40 of 46 posts

Re: Linux syslog on the way out?

#31
I'd love to see something like that implemented - for the benefit of having semi-structured messages... Oh wait, there are lots of daemons which already do that. And they use properly documented formats, like GELF. And they support forwarding over network. And they store data inside a proper database, instead of yet-another-file-format.

I'd much rather they simply blessed one of the existing solutions (greylog2, fluent, loads of others) and created a compatible minimal daemon which can be included by default in a distribution. Currently they're neither providing new good solution, nor solving the structured log problem (flat k/v? why not just go json/bson?). They're also trying to solve single-host security, which seems pretty close to impossible without moving logging into something like TPM - just send the logs outside, so that the logging channel is append only.

There is also one point which makes me really worried about what the hell are they thinking... "The timestamps generally do not carry timezone information, even though some newer specifications define support for it." Simple solution is to log in UTC, always, everywhere. Run your servers in UTC too - unless it's your local desktop, there's no reason not to.

Re: Linux syslog on the way out?

#32
post #29

Earlier quoted context omitted.

What exactly was the argument against NTP?

They weren't sure it was good enough, or good any more, and they were trying to figure out ways of maintaining a timebase completely within the application, from server to server. These were developers, both salary and contract, in their 30s. As a former sysadmin, now programmer, it was surreal.

It's sad that this is basically a microcosm of the way things typically are in our field.

Re: Linux syslog on the way out?

#33
post #21

Earlier quoted context omitted.

What are a few examples of other "old *nix tools" that are showing their age and have "known deficiencies?"

CVS?

I believe this is a bad example. Because CVS is only under maintenance there is no new feature getting added and the last stable release was in 2008. Also anyone that does a bit of research on version control software would realize CVS is only around for legacy purposes and there is enough paths to convert to SVN or a more "newish" DVCS.

Re: Linux syslog on the way out?

#34
Mm. So many nails in this coffin.

Many times I've had a need to grep or tail a log in a hurry. This kind of change compromises the entire modus operandi of good Unix admins.

There must be tens of thousands of Unix-based tools out there that use the syslog(3) interface and I'm damn sure that most of them will not change Just Because Someone At Fedora wants them to.

I don't believe a binary format is any more inherently secure than a textual one. Want non-repudiation of logs? Sign them and send them off-box.

They would be much smarter to introduce incremental improvements i.e. extend the existing syslog protocol to accept a structured data format; introduce a log signing journal for integrity checks.

Re: Linux syslog on the way out?

#35
post #10

Earlier quoted context omitted.

Cryptography, properly applied, allows you to verify file integrity . The real problem becomes one of key management however -- if the Journal proposal solves this issue (and it's not a simple one) then it's possible that the "has the integrity of my system logs been preserved/maintained" question can be answered. Managing keys properly is highly non-trivial, especially on a system in which you want those keys to be…

Cryptography allows you to verify that somebody who does not possess your private key did not modify the file. A hacker on the same system as a syslogger has the private key of the syslogger. They can "simply" (for varying values of simply) replay the logs again, only with whatever events they want filtered out or modified, and it will pass the crypto checks just fine. You have to offload your logs somewhere else, at…

Using PKI, this isn't necessarily true. Even constructing a system with simple one-way hashes might be sufficient.

If the logging system is based on a hash of the current and prior records, you get chaining and integrity validation prior to the attack (one problem of attacks is wiping out immediate or all prior history). If you've got remote storage / logging, particularly on multiple systems with independent trust models (compromising system A doesn't necessitate compromise of B and/or C), you're in better shape still.

A signature which required periodic updating of a key from a key provider via a mechanism that prevents later key reuse would mean that past records couldn't be fabricated or modified after the fact. This might require passing data through two or more systems to accomplish the signing in a trusted fashion (not particularly amenable to high-rate logging).

The verification key need not be on the logging system at all (and ideally wouldn't be). You'd access and verify logs from a standalone hardened and highly trusted system (say: booted from known good static media).

Much of which is good for the extreme case, but as I note in my other comments, isn't particularly practical for day-to-day needs. Which means it's also likely to be unfamiliar to technical staff, rarely practiced, rarely exercised, and buggy.

Re: Linux syslog on the way out?

#36
post #21
post #11

Earlier quoted context omitted.

I disagree. A lot of these old *nix tools are showing their age and could use a refresh based on known deficiencies. Creating new tools won't harm the old ones so I don't see a problem.

What are a few examples of other "old *nix tools" that are showing their age and have "known deficiencies?"

Maybe cron? See http://www.fi.muni.cz/~kas/blog/index.cgi/computers/rethinki...

Re: Linux syslog on the way out?

#37
post #21
post #11

Earlier quoted context omitted.

I disagree. A lot of these old *nix tools are showing their age and could use a refresh based on known deficiencies. Creating new tools won't harm the old ones so I don't see a problem.

What are a few examples of other "old *nix tools" that are showing their age and have "known deficiencies?"

init / sysvinit / upstart / systemd: dependencies, reliable process monitoring & restarts, parallelization.

cron: take your pick, but I'd like a scheduler which would produce useful email / messaging reports when things go wrong, and shut the fuck up when they don't, while logging activity in markedly more detail than current jobs do. To accomplish this today you'd need to follow a strict template on your actual cronjobs. Stuff like: what host am I on, what crontab am I being run out of, what user am I running as, what script(s) / program(s) am I invoking.

nfs: performance, reliability, security, arbitrary group limits.

Generally: performance monitoring. We're still relying on top, uptime, vmstat, iostat, strace, ltrace, (sometimes systemtap/dtrace), wireshark, netstat, lsof, and related tools. There are a few new tricks coming out, and some ways to integrate data from multiple sources, but often tuning systems (especially where VMs or application engines are concerned) resembles a seance / archaeological dig / bro knowledge / myth / wishful thinking / Republican plank. For the love of Dennis, it's been 40 years, give me some goddamned insights.

There are probably a bunch of others, though I'm trying to stick with basics that you might find in The UNIX Programming Environment or similar -- really foundational basics (or reasonable extensions of same).

The barrier to change is that most of this stuff works well enough most of the time, and too much depends on it. I'm seeing a lot of exploration of alternatives around some core systems though, and suspect we'll see significant changes in the next 5-10 years. Possibly even sooner, though history suggests slower adoption.

Re: Linux syslog on the way out?

#38
post #10

Earlier quoted context omitted.

Cryptography allows you to verify that somebody who does not possess your private key did not modify the file. A hacker on the same system as a syslogger has the private key of the syslogger. They can "simply" (for varying values of simply) replay the logs again, only with whatever events they want filtered out or modified, and it will pass the crypto checks just fine. You have to offload your logs somewhere else, at…

Using PKI, this isn't necessarily true. Even constructing a system with simple one-way hashes might be sufficient. If the logging system is based on a hash of the current and prior records, you get chaining and integrity validation prior to the attack (one problem of attacks is wiping out immediate or all prior history). If you've got remote storage / logging, particularly on multiple systems with independent trust m…

If they are on the box and you don't have your logs offloaded, they can at the very least destroy the logs, and barring your complicated key updating system which would itself require lots of careful review, probably forge anything they want. But destruction is a pretty useful capability itself.

If the logs are being offloaded off the system, then you don't need any of this either because they are already being put somewhere the hacker can't touch. (Of course if they hack that system then you're in real trouble, but that's just moving the problem around; ultimately if your hacker owns everything, you've really, really lost.)

There's no inbetween state where all this complexity solves any problem. Either you've got off-box storage and you don't need it, or you don't and you've already lost if a hacker gets root.

Re: Linux syslog on the way out?

#39
They're going to need really good import and export tools. If they can provide those, then most of the objections to the change go away. If they can't, then I doubt it will gain traction.

Re: Linux syslog on the way out?

#40
post #38

Earlier quoted context omitted.

Using PKI, this isn't necessarily true. Even constructing a system with simple one-way hashes might be sufficient. If the logging system is based on a hash of the current and prior records, you get chaining and integrity validation prior to the attack (one problem of attacks is wiping out immediate or all prior history). If you've got remote storage / logging, particularly on multiple systems with independent trust m…

If they are on the box and you don't have your logs offloaded, they can at the very least destroy the logs, and barring your complicated key updating system which would itself require lots of careful review, probably forge anything they want. But destruction is a pretty useful capability itself. If the logs are being offloaded off the system, then you don't need any of this either because they are already being put s…

I agree with your points here.

I don't see the proposed journal solution really solving on-system log integrity.

There are a few other points (processes impersonating other processes, e.g., or extended logging formats) which might be better supported in something other than a traditional syslog. But I'm absolutely not sold on Journal.

Post reply on HN