Grepping logs is still terrible • Hacker News Board

Re: Grepping logs is still terrible

#121

Post by kbenson » Fri, May 08, 2015, 3:49 PM UTC

crdoconnor wrote: post #120

kbenson wrote: post #100

Earlier quoted context omitted.

I have no problem with json output in log files, but I would greatly prefer it be constrained to the message portion of a logline. At a minimum I generally want three things per line, a timestamp (in ISO 8601 or something close), a message type (info, warning, error, etc) or log entry source, and the message itself. I don't want to be looking into the JSON structure itself for a timestamp, especially when the field e…

>I don't want to be looking into the JSON structure itself for a timestamp A) JSON parsers are relatively common and reliable. B) The timestamp would be human readable even without the parser. >especially when the field encoding the timestamp may be called something slightly different based on what generated the log... I often come across logs that put timestamps in different places on the line and encode them differ…

My point is really around having the date be in a well defined place that isn't necessarily defined by the application that's logging. If the log entry date is at the beginning of the line, there's no ambiguity as to whether it's the log entry date or some other date being logged, and it also doesn't require parsing the JSON at all to filter by the date. If it's not at some very standard location that's easy to filer by (a possibly changing JSON property does not qualify), they it's hard to know you are filtering on the right data, and may also require transform before filtering. JSON parsers are fast. Multi-GB log files will still cause some extra overhead and slow the operation down, so it's best to reduce the working set before parsing the JSON.

Re: Grepping logs is still terrible

#122

Post by crdoconnor » Sat, May 09, 2015, 9:10 AM UTC

kbenson wrote: post #121

crdoconnor wrote: post #120

Earlier quoted context omitted.

>I don't want to be looking into the JSON structure itself for a timestamp A) JSON parsers are relatively common and reliable. B) The timestamp would be human readable even without the parser. >especially when the field encoding the timestamp may be called something slightly different based on what generated the log... I often come across logs that put timestamps in different places on the line and encode them differ…

My point is really around having the date be in a well defined place that isn't necessarily defined by the application that's logging. If the log entry date is at the beginning of the line, there's no ambiguity as to whether it's the log entry date or some other date being logged, and it also doesn't require parsing the JSON at all to filter by the date. If it's not at some very standard location that's easy to filer…

>My point is really around having the date be in a well defined place that isn't necessarily defined by the application that's logging. If the log entry date is at the beginning of the line, there's no ambiguity as to whether it's the log entry date or some other date being logged, and it also doesn't require parsing the JSON at all to filter by the date.

Take this example:

1-1-15 1:1:1 Info Log message A

12-13-15 12:34:55 Debug Log message B

12-13-15 1:34:55 Error log message C

12-13-15 1:34:55Error log message D

[12-13-15 1:34:55]Error log message E

It doesn't require parsing JSON to get the date, you're right about that. It's harder than parsing JSON, though.

Re: Grepping logs is still terrible

#123

Post by gambiter » Sat, May 09, 2015, 11:58 PM UTC

nailer wrote: post #88

gambiter wrote: post #57

Earlier quoted context omitted.

'tooling gets deleted' could easily happen after changing logging systems... while it would be shortsighted to uninstall your old logging system entirely (if you have logs laying around in that format) it's not unheard of. The more likely situation would be that the logs are stored on a shared storage server, and the machine you are using to look at the logs doesn't have the logging system installed.

> The more likely situation would be that the logs are stored on a shared storage server, and the machine you are using to look at the logs doesn't have the logging system installed. So expose the shared storage to a system running any current mainstream Linux distribution. I understand what you're saying, but this still doesn't seems like a huge concern.

... We were talking about logging systems with proprietary tools for manipulating logs. Ergo, 'any current mainstream Linux distribution' wouldn't have them installed by default.

Re: Grepping logs is still terrible

#124

Post by kbenson » Mon, May 11, 2015, 7:56 PM UTC

crdoconnor wrote: post #122

kbenson wrote: post #121

Earlier quoted context omitted.

My point is really around having the date be in a well defined place that isn't necessarily defined by the application that's logging. If the log entry date is at the beginning of the line, there's no ambiguity as to whether it's the log entry date or some other date being logged, and it also doesn't require parsing the JSON at all to filter by the date. If it's not at some very standard location that's easy to filer…

>My point is really around having the date be in a well defined place that isn't necessarily defined by the application that's logging. If the log entry date is at the beginning of the line, there's no ambiguity as to whether it's the log entry date or some other date being logged, and it also doesn't require parsing the JSON at all to filter by the date. Take this example: 1-1-15 1:1:1 Info Log message A 12-13-15 12…

Note two replies of mine prior where I state ISO 8601 or similar. Also not where I said the json would be constrained to the message portion of the entry. Preferably there's a logging mechanism that takes care of that for you, so you can't screw up the timestamp and type portions of the entry. In that case, your entries become:

  2015-01-01 01:01:01 Info Log message A
  2015-12-13 12:34:55 Debug Log message B
  2015-12-13 01:34:55 Error log message C
  2015-12-13 13:34:55 Error log message D # let's assume that was 1 PM data for the sake of the example
  2015-12-13 01:34:55 Error log message E

Getting the date is trivial. Getting the type is also trivial. Give a static field size to type and it's event more so. The point is, you abstract the message from the rest of it, so the message can't screw up the metadata of the entry, and log whatever you want for the actual message (xml, json, plain text, whatever, just no raw newlines). This is what we have today with syslog, sans newline replacement and a slightly different date format (but still unambiguous). It works. It's useful. It's VERY easy to filter type type or date. You can take the first X chars and split on space/whitespace if you need to. You can log a message of a few megabytes and if there's no raw newlines there's efficient utilities to ignore that until you have what you want (/bin/cut).