Lnav – An advanced log file viewer for the small-scale • Hacker News Board

Re: Lnav – An advanced log file viewer for the small-scale

#51

Post by LinuxBender » Thu, Jan 05, 2023, 9:31 PM UTC

electroly wrote: post #34

LinuxBender wrote: post #32

Earlier quoted context omitted.

I've also seen people say that and to instead use "blkdiscard -sz" but that also does not write to specific blocks. I still can not find a definitive source saying that shred is not able to write to specific blocks on a SSD and it is not clear to me that wear leveling actually prevents shred from writing to specific blocks. But I agree that I have seen people say this repeatedly on StackExchange, ServerFault and Redd…

Wear leveling prevents anyone from writing to specific blocks. It decouples the OS's block numbering from the physical blocks. That indirection allows it to distribute writes evenly across the drive even if the OS is writing to consecutive blocks (or the same block). This behavior is fundamental; it's what wear leveling is and why it exists. Wear leveling exists to prevent you from writing to the same block repeatedl…

So my concern with following this logic is that such tools that are writing to specific block locations would then be wiping unrelated inodes depending on the low level logic. I tried emailing Colin but his last known email is invalid now. He may be working for the NSA so I probably won't be able to reach him. Some people there are responsive, some are not and a few are on here.

So even if wear leveling prevents overwriting a file then such tools should in theory be a risk of data corruption. If this is the case then the tool should be updated to detect if the target is an SSD and abort with a scary message. Perhaps another route is to reach out to the coreutils team.

Re: Lnav – An advanced log file viewer for the small-scale

#52

Post by j0057 » Thu, Jan 05, 2023, 10:39 PM UTC

LinuxBender wrote: post #32

electroly wrote: post #27

Earlier quoted context omitted.

Shred doesn't work at all on SSDs because wear leveling will spray the writes all over the drive, rather than overwriting the blocks you intended to. Your random bytes will end up in new blocks, not overwriting the original blocks. Trying to shred individual files is a totally pointless exercise on SSDs. It isn't doing what you want it to do.

I've also seen people say that and to instead use "blkdiscard -sz" but that also does not write to specific blocks. I still can not find a definitive source saying that shred is not able to write to specific blocks on a SSD and it is not clear to me that wear leveling actually prevents shred from writing to specific blocks. But I agree that I have seen people say this repeatedly on StackExchange, ServerFault and Redd…

It's also explained in the shred manual itself, that seems like a decently authoritative source to me: https://www.gnu.org/software/coreutils/manual/html_node/shre... — halfway the page all the caveats are listed up to & including forensic analysis of magnetic traces of the deleted data.

Re: Lnav – An advanced log file viewer for the small-scale

#53

Post by joshka » Fri, Jan 06, 2023, 2:21 AM UTC

zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Take a look at the docs linked on the main page of Serilog[1] and the various videos[2] and blog posts[3] by the devs who make Seq.

[1]: https://serilog.net/

[2]: https://datalust.co/

[3]: https://blog.datalust.co/

Re: Lnav – An advanced log file viewer for the small-scale

#54

Post by nickcox » Fri, Jan 06, 2023, 4:09 AM UTC

zulu-inuoe wrote: post #12

Say I'm building a logging library, is there any sort of generally agreed-upon standard per what a log file should look like? I know there's several formats that lnav supports, but I'm not familiar with them. I've always had trouble before figuring out what a good compromise between log format and flexibility looks like. Especially wrt newlines in the log message itself (eg if I want to log a stack trace) Any thought…

Structured logs. Probably json.

Re: Lnav – An advanced log file viewer for the small-scale

#55

Post by 10987654321 » Fri, Jan 06, 2023, 9:48 AM UTC

zulu-inuoe wrote: post #47

10987654321 wrote: post #13

Earlier quoted context omitted.

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

Thank you! Now when you say json logging, are there any common patterns you can guide me to look at? Just looking around for JSON logging gives me results like [1] which talk about JSONL (object per line, presumably JSON serializer makes sure to not emit literal newlines). But some other places describe this a bit more liberally. And [2] notes that I should include a time stamp in each log entry which makes sense. [1…

In general if you use things like filebeat or promtail to do log ingestion into centralized log search systems (elasticsearch or loki in these examples) they prefer one object per line.

It makes parsing and keeping track of the "state" of the file a lot easier. Say that your application crashes/gets killed halfway through writing a log message / json dict and then gets restarted and appends to the log file. How should the log reader handle that case if it suddenly becomes a valid nested object? And even if it doesn't, should it throw away the first new log message as well because that was embedded in the invalid json object? Much easier to just say "one line is one json object, if there are literal newlines that's the delimeter to start a new parse".

And yes in any case it's good to have a timestamp on your log message no matter the format, unless you're logging somewhere you know that it gets added immediately (like the systemd journal). Your log parser/forwarder can add a timestamp for when it reads your log message but that is not necessarily the same as when your application emits it.

Re: Lnav – An advanced log file viewer for the small-scale

#56

Post by 10987654321 » Fri, Jan 06, 2023, 10:10 AM UTC

LinuxBender wrote: post #51

electroly wrote: post #34

Earlier quoted context omitted.

Wear leveling prevents anyone from writing to specific blocks. It decouples the OS's block numbering from the physical blocks. That indirection allows it to distribute writes evenly across the drive even if the OS is writing to consecutive blocks (or the same block). This behavior is fundamental; it's what wear leveling is and why it exists. Wear leveling exists to prevent you from writing to the same block repeatedl…

So my concern with following this logic is that such tools that are writing to specific block locations would then be wiping unrelated inodes depending on the low level logic. I tried emailing Colin but his last known email is invalid now. He may be working for the NSA so I probably won't be able to reach him. Some people there are responsive, some are not and a few are on here. So even if wear leveling prevents over…

> writing to specific block locations would then be wiping unrelated inodes depending on the low level logic

It won't wipe unrelated files as far as the file system sees it, but it might overwrite some previously discarded internal blocks. "inodes" is a too high-level concept in this context.

You can read more about it on wikipedia which might be an authorative enough source for you? https://en.wikipedia.org/wiki/Wear_leveling

Also note that most ssds actually have more internal blocks available than what they present to the host device so that they can have a "cache" to be able to move things around internally, and also so they can mark certain positions as "bad" when writes to one internal block start failing and still operate properly.

Re: Lnav – An advanced log file viewer for the small-scale

#57

Post by 10987654321 » Fri, Jan 06, 2023, 10:41 AM UTC

zulu-inuoe wrote: post #47

10987654321 wrote: post #13

Earlier quoted context omitted.

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

Thank you! Now when you say json logging, are there any common patterns you can guide me to look at? Just looking around for JSON logging gives me results like [1] which talk about JSONL (object per line, presumably JSON serializer makes sure to not emit literal newlines). But some other places describe this a bit more liberally. And [2] notes that I should include a time stamp in each log entry which makes sense. [1…

Another benefit with json/logfmt that bears mentioning explicitly: it has structure.

This means that you shouldn't just write (to reuse the previous example):

    msg="Request for brandur@mutelight.org finished with status 200"

you should do it like

    msg="Request finished" status=200 user=brandur@mutelight.org

and not put any variables into the msg key (and not really do advanced formatting for any of the keys for that matter). This way once you get it put into a log system that understands your format you can do searches like "all log messages where user=foo" or "all statuses that are >=500 and <600" or search on specific messages, all without having to craft elaborate regular expressions and with better performance since the log search system can do indexing and various optimizations so that it doesn't have to be a full-text search every time.

Re: Lnav – An advanced log file viewer for the small-scale

#58

Post by 10987654321 » Fri, Jan 06, 2023, 10:50 AM UTC

tstack wrote: post #24

10987654321 wrote: post #13

Earlier quoted context omitted.

Not sure about lnav but most log aggregation systems support json and logfmt-formatted logs, and there are many standard logging libraries that supports emitting those formats Of those json is better if you want to be able to do more advanced stuff (nest dictionaries, use lists, ...) and logfmt is better if you want to have human-readable logs without external tools as well, an example line can look like msg="Request…

lnav does support JSON-lines and logfmt logs. For JSON-lines, it will pretty-print the log messages to make them human readable. For logfmt, I seem to remember the spec not being very clear on quoting semantics (maybe I'm wrong). Anyhow, I would suggest using JSON since it has pretty broad support at this point.

Hmmm, it doesn't say in https://lnav.org/features#automatic-log-format-detection but i see that at least json (and xml) is mentioned under the pretty-printing header.

You would know what is supported what with you being the author, just saying that the docs aren't super clear from a quick glance :).

And yes, I second the suggestion to focus on JSON. The main benefit of logfmt is that it's simpler for a human to parse directly but in general you probably shouldn't aim for that so..

Re: Lnav – An advanced log file viewer for the small-scale

#59

Post by tstack » Fri, Jan 06, 2023, 5:15 PM UTC

10987654321 wrote: post #58

tstack wrote: post #24

Earlier quoted context omitted.

lnav does support JSON-lines and logfmt logs. For JSON-lines, it will pretty-print the log messages to make them human readable. For logfmt, I seem to remember the spec not being very clear on quoting semantics (maybe I'm wrong). Anyhow, I would suggest using JSON since it has pretty broad support at this point.

Hmmm, it doesn't say in https://lnav.org/features#automatic-log-format-detection but i see that at least json (and xml) is mentioned under the pretty-printing header. You would know what is supported what with you being the author, just saying that the docs aren't super clear from a quick glance :). And yes, I second the suggestion to focus on JSON. The main benefit of logfmt is that it's simpler for a human to parse…

> Hmmm, it doesn't say in https://lnav.org/features#automatic-log-format-detection but i see that at least json (and xml) is mentioned under the pretty-printing header.

Yes, I should mention it on the features page. It's currently only mentioned in the main docs:

https://docs.lnav.org/en/latest/formats.html

Re: Lnav – An advanced log file viewer for the small-scale

#60

Post by [deleted] » Tue, Jan 10, 2023, 10:39 AM UTC

electroly wrote: post #27

LinuxBender wrote: post #18

Earlier quoted context omitted.

It would be as hard as writing the file {n} number of times again. So if one created a 1MB log file and then did a 3 pass wipe that would write 3MB. Number of passes would really just depend on the system being worked on. If PCI I would expect 7 passes and I am sure that would just be factored into the cost of the servers for that environment. Shred also isn't perfect as it has no concept of the file systems journal…

Shred doesn't work at all on SSDs because wear leveling will spray the writes all over the drive, rather than overwriting the blocks you intended to. Your random bytes will end up in new blocks, not overwriting the original blocks. Trying to shred individual files is a totally pointless exercise on SSDs. It isn't doing what you want it to do.

[deleted]