Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lnav – An advanced log file viewer for the small-scale

lnav.org

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Lnav – An advanced log file viewer for the small-scale

#21

Post by pinkorchid »

LinuxBender wrote: post #5

The package is available in Alpine so I tried it out. It seems to work best if the terminal is using en_US.UTF-8. My default is C but I have a function to set UTF-8. I was curious if it could handle stdin and it does. It saves the output in ~/.config/lnav/ something to probably add to my clean up script. It is convenient that it can prepend timestamps to logs. I could see this being nice for people that do QA on buil…

This will depend on the specific threat model for the sensitive data you are dealing with, but generally files in an SSD can't be destroyed securely (and shred specifically does not help due to wear leveling).

Instead, you may want to encrypt your disk so that as soon as the key is gone it all becomes unreadable. For a bigger threat, maybe you need to follow the NIST destroy guidelines [1] to "Disintegrate, Pulverize, Melt, and Incinerate" the media.

1: https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.s...

Re: Lnav – An advanced log file viewer for the small-scale

#22

Post by tstack »

LinuxBender wrote: post #5

The package is available in Alpine so I tried it out. It seems to work best if the terminal is using en_US.UTF-8. My default is C but I have a function to set UTF-8. I was curious if it could handle stdin and it does. It saves the output in ~/.config/lnav/ something to probably add to my clean up script. It is convenient that it can prepend timestamps to logs. I could see this being nice for people that do QA on buil…

The data consumed from stdin is stored in a file to support scrollback.

> something to probably add to my clean up script.

These capture files are deleted if they are older than a day the next time lnav is run.

(I realize they should've been stored in ~/.local/state instead of ~/.config ...)

Re: Lnav – An advanced log file viewer for the small-scale

#23

Post by jteppinette »

10987654321 wrote: post #13
zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

Agree with logfmt. I wrote the logfmter python library: https://github.com/jteppinette/python-logfmter. You can quickly have all of your logs (including 3rd party) converted to this style.

Re: Lnav – An advanced log file viewer for the small-scale

#24

Post by tstack »

10987654321 wrote: post #13
zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

lnav does support JSON-lines and logfmt logs. For JSON-lines, it will pretty-print the log messages to make them human readable.

For logfmt, I seem to remember the spec not being very clear on quoting semantics (maybe I'm wrong). Anyhow, I would suggest using JSON since it has pretty broad support at this point.

Re: Lnav – An advanced log file viewer for the small-scale

#25

Post by ithkuil »

10987654321 wrote: post #13
zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

Has anybody written an actual spec for logfmt? I noticed that different implementations handle escaping of quoted strings subtly differently

Re: Lnav – An advanced log file viewer for the small-scale

#27

Post by electroly »

LinuxBender wrote: post #18
c0wb0yc0d3r wrote: post #16

Earlier quoted context omitted.

I was under the impression that the shred command was pretty hard on SSDs. Did that change?

It would be as hard as writing the file {n} number of times again. So if one created a 1MB log file and then did a 3 pass wipe that would write 3MB. Number of passes would really just depend on the system being worked on. If PCI I would expect 7 passes and I am sure that would just be factored into the cost of the servers for that environment. Shred also isn't perfect as it has no concept of the file systems journal…

Shred doesn't work at all on SSDs because wear leveling will spray the writes all over the drive, rather than overwriting the blocks you intended to. Your random bytes will end up in new blocks, not overwriting the original blocks. Trying to shred individual files is a totally pointless exercise on SSDs. It isn't doing what you want it to do.

Re: Lnav – An advanced log file viewer for the small-scale

#28

Post by GordonS »

10987654321 wrote: post #13
zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

You can use logfmt with Serilog on dotnet too:

https://github.com/serilog-contrib/Serilog.Logfmt

Post reply on HN