Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoLog – a nanosecond scale logging system for C++

github.com

Post reply on HN
Print view View original

21–30 of 67 posts

Re: NanoLog – a nanosecond scale logging system for C++

#21

Post by detaro »

gpderetta wrote: post #18
detaro wrote: post #16

Earlier quoted context omitted.

If I remember correctly, it is fine if there actually was an object of that type (or a "similar" type) in that location. So casting to char* and back is fine, casting to char and then to int-type to inspect multiple bytes is fine as long as alignment plays correctly, ... It seems like the function only cast to T* if the input pointer hasn't been changed (the paths that do modify it return early), so there's a value t…

> casting to char and then to int-type to inspect multiple bytes is fine as long as alignment plays correctly It is only fine if there were the same exact int types at that address. Remember, UB usually isn't the reinterpret_cast but the dereference.

I thought the int-types had a special exception making that always possible (if there's a bunch of bytes, you can turn them into an int), but I could very well be wrong on that detail.

Re: NanoLog – a nanosecond scale logging system for C++

#22

Post by rafa1981 »

Author of mini-async-log and mini-async-log-c here.

https://github.com/RafaGago/mini-async-log

https://github.com/RafaGago/mini-async-log-c

I wrote both some time ago and AFAIK they were the fastest, how does nanolog compare against them?

At least before the source file preprocessor Nanolog was slower.

Here I had a benchmark project with Nanolog on an old version, maybe you can PR an update:

https://github.com/RafaGago/logger-bench

Re: NanoLog – a nanosecond scale logging system for C++

#23

Post by rafa1981 »

bt848 wrote: post #17
rafa1981 wrote: post #14

Earlier quoted context omitted.

Author of mini-async-log and its C variant here. Some time ago I wrote a benchmark for different loggers, but it uses an old version of Nanolog. https://github.com/RafaGago/logger-bench

Neat. Your code actually contains the issue I was curious about. Why do you log at level ERROR? As I understand it this causes glog to synchronize the output stream after every message.

Lack of knowledge on glog

Re: NanoLog – a nanosecond scale logging system for C++

#24

Post by rafa1981 »

bt848 wrote: post #17
rafa1981 wrote: post #14

Earlier quoted context omitted.

Author of mini-async-log and its C variant here. Some time ago I wrote a benchmark for different loggers, but it uses an old version of Nanolog. https://github.com/RafaGago/logger-bench

Neat. Your code actually contains the issue I was curious about. Why do you log at level ERROR? As I understand it this causes glog to synchronize the output stream after every message.

But worst case latency, the most interesting property of an async logger is still unaffected I guess...

Re: NanoLog – a nanosecond scale logging system for C++

#25

Post by gpderetta »

muststopmyths wrote: post #3

I spent a lot of time trying to build a fast logging system in my last couple of jobs. The basic lesson (and I'm only talking about C/C++/C# here) is that you will spend most of your time formatting strings if you do your file I/O asynchronously. Since this system has a preprocessor mode, I assume they learnt the same lesson. The bigger lesson is that it really doesn't matter how many millions of logs you can generat…

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously.

That's why you are supposed to do your formatting in the background thread :).

> doesn't matter how many millions of logs you can generate per second

Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal overhead on the surrounding code.

Re: NanoLog – a nanosecond scale logging system for C++

#26

Post by Leszek »

MauranKilom wrote: post #9
Leszek wrote: post #7

Earlier quoted context omitted.

You'd think so, but in fact reinterpret_cast from char* (and unsigned char , and std::byte ) is explicitly allowed by the type aliasing rules.

To the best of my knowledge, casting to char* is totally fine (inspecting an object as bytes) under certain constraints. What is not fine is pretending that an object lives at a position in memory where it does not (i.e. treating bytes of memory as some object through a reinterpret_cast away from char* ). Edit: To clarify, casting away from char* is of course allowed if you cast to whatever object type actually lives…

I had thought that, as long as the object in that location is valid (e.g. constructed with placement new and not since destructed, with the `char` as a pointer to the storage), then accessing it through the `char` is valid -- but maybe that's only true for `void*` now that I think about it. Either way, fair enough, placement new probably hasn't happened (haven't read the rest of the code).

Re: NanoLog – a nanosecond scale logging system for C++

#27

Post by [deleted] »

nemetroid wrote: post #20

From a quick reading of the paper, it sounds like Nanolog is internally translating logging messages into a compacted format, which must be further processed to become human readable. This further processing is not included in the benchmark. section 4.3, "Decompressor/Aggregator" > The final component of the NanoLog system is the decompressor/aggregator, which takes as input the compacted log file generated by the ru…

[deleted]

Re: NanoLog – a nanosecond scale logging system for C++

#28

Post by usefulcat »

The description says "the compilation builds a version of the NanoLog library that is non-portable, even between compilations of the same application". They also mention that a decompressor application also gets built and can be used to reconstitute a human-readable version of the log file. My question is, is the decompressor application also tightly coupled to a particular compilation? It sounds like the answer is yes, but I'd really like to be wrong about that.

That would really suck for cases where you want to store old log files for later use. You'd have to also store a copy of the decompressor program that corresponds to each log file. Or you'd have to convert to human-readable format before storing the log files, which loses a number of the benefits of the binary log format (compact size, faster to parse).

Re: NanoLog – a nanosecond scale logging system for C++

#29

Post by Const-me »

Recently built similar stuff. Even using same __rdtsc() for time. I didn't require any preprocessing. Instead, I require the strings come from readonly section of a module, and logging pointer values. Also logging into circular buffer in shared memory instead of file, as I was only interested in knowing what the app did immediately before a rare crash which takes hours to reproduce.

Re: NanoLog – a nanosecond scale logging system for C++

#30

Post by nemetroid »

usefulcat wrote: post #28

The description says "the compilation builds a version of the NanoLog library that is non-portable, even between compilations of the same application". They also mention that a decompressor application also gets built and can be used to reconstitute a human-readable version of the log file. My question is, is the decompressor application also tightly coupled to a particular compilation? It sounds like the answer is y…

In the related paper[1], this paragraph (5.1.2) strongly suggests that the decompressor is non-portable as well:

> NanoLog performs best when there is little dynamic information in the log message. This is reflected by staticString, a static message, in the throughput benchmark. Here, NanoLog only needs to output about 3-4 bytes per log message due to its compaction and static extraction techniques.

1: https://www.usenix.org/conference/atc18/presentation/yang-st...

Post reply on HN