Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoLog – a nanosecond scale logging system for C++

github.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: NanoLog – a nanosecond scale logging system for C++

#31

Post by gpderetta »

detaro wrote: post #21
gpderetta wrote: post #18

Earlier quoted context omitted.

> casting to char and then to int-type to inspect multiple bytes is fine as long as alignment plays correctly It is only fine if there were the same exact int types at that address. Remember, UB usually isn't the reinterpret_cast but the dereference.

I thought the int-types had a special exception making that always possible (if there's a bunch of bytes, you can turn them into an int), but I could very well be wrong on that detail.

Char is the only exception.

And now std::byte IIRC

Re: NanoLog – a nanosecond scale logging system for C++

#32

Post by rafa1981 »

Const-me wrote: post #29

Recently built similar stuff. Even using same __rdtsc() for time. I didn't require any preprocessing. Instead, I require the strings come from readonly section of a module, and logging pointer values. Also logging into circular buffer in shared memory instead of file, as I was only interested in knowing what the app did immediately before a rare crash which takes hours to reproduce.

I did exactly the same on my two loggers, as a C literal is always read only and unique, passing a printf-like string is passing a pointer.

The it's just a matter of adding validation and passing the arguments in a compact way.

Re: NanoLog – a nanosecond scale logging system for C++

#33

Post by Matthias247 »

gpderetta wrote: post #25
muststopmyths wrote: post #3

I spent a lot of time trying to build a fast logging system in my last couple of jobs. The basic lesson (and I'm only talking about C/C++/C# here) is that you will spend most of your time formatting strings if you do your file I/O asynchronously. Since this system has a preprocessor mode, I assume they learnt the same lesson. The bigger lesson is that it really doesn't matter how many millions of logs you can generat…

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously. That's why you are supposed to do your formatting in the background thread :). > doesn't matter how many millions of logs you can generate per second Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal…

> That's why you are supposed to do your formatting in the background thread :).

Which then has the drawback that every argument that gets transferred to the background thread must first be heap allocated or static - instead of only the final buffer being that. My gut feeling is that the additional allocation and memory management costs typically outweigh the cost of formatting in the current thread. Even if one pools all logging buffers less buffers are needed for transferring only formatted buffers.

Re: NanoLog – a nanosecond scale logging system for C++

#34

Post by bonzini »

MauranKilom wrote: post #15
Iwan-Zotow wrote: post #8

Earlier quoted context omitted.

char* is universal memory format, you could specialize from that into anything you want

That's not how the C++ object model works. Objects in C++ have clearly defined lifetimes. If you reinterpret_cast (someCharBuffer) and then dereference that, you have Undefined Behavior unless an int object is alive at that exact location. You can do int value[2] = {0, 0}; char* ptr = reinterpret_cast (value) + sizeof(int); (*reinterpret_cast (ptr))++; or (given knowledge about compiler padding): struct XY { double x…

You cannot do the last unless you know that the argument of placement new is suitably aligned to receive an int object, however.

Re: NanoLog – a nanosecond scale logging system for C++

#35

Post by bonzini »

Const-me wrote: post #29

Recently built similar stuff. Even using same __rdtsc() for time. I didn't require any preprocessing. Instead, I require the strings come from readonly section of a module, and logging pointer values. Also logging into circular buffer in shared memory instead of file, as I was only interested in knowing what the app did immediately before a rare crash which takes hours to reproduce.

How do you deal with ASLR (logging from shared libraries or from position-independent executables)?

Re: NanoLog – a nanosecond scale logging system for C++

#36

Post by gpderetta »

Matthias247 wrote: post #33
gpderetta wrote: post #25

Earlier quoted context omitted.

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously. That's why you are supposed to do your formatting in the background thread :). > doesn't matter how many millions of logs you can generate per second Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal…

> That's why you are supposed to do your formatting in the background thread :). Which then has the drawback that every argument that gets transferred to the background thread must first be heap allocated or static - instead of only the final buffer being that. My gut feeling is that the additional allocation and memory management costs typically outweigh the cost of formatting in the current thread. Even if one pool…

Normally you would copy the arguments to be formatted. This is usually less bytes than the formatted output.

Re: NanoLog – a nanosecond scale logging system for C++

#37

Post by cma »

gpderetta wrote: post #25
muststopmyths wrote: post #3

I spent a lot of time trying to build a fast logging system in my last couple of jobs. The basic lesson (and I'm only talking about C/C++/C# here) is that you will spend most of your time formatting strings if you do your file I/O asynchronously. Since this system has a preprocessor mode, I assume they learnt the same lesson. The bigger lesson is that it really doesn't matter how many millions of logs you can generat…

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously. That's why you are supposed to do your formatting in the background thread :). > doesn't matter how many millions of logs you can generate per second Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal…

Shared memory to a background process is better, in that it can survive most classes of process crash.

Re: NanoLog – a nanosecond scale logging system for C++

#38

Post by gpderetta »

MauranKilom wrote: post #5

The code overall is pretty clever, but at the core of all this they completely ignore strict aliasing to dump stuff into a char* buffer indiscriminately... Look at this function: https://github.com/PlatformLab/NanoLog/blob/master/runtime/N... T argument = *reinterpret_cast (*in); ... uint32_t stringBytes = *reinterpret_cast (*in); A total of 6 reinterpret_casts in that file alone. I didn't see any indication that you…

> Adding a volatile to fix a threading bug is just a big no-no.

The fence instructions also don't make much sense. Not sure why one would not use std::atomic in a c++17 only library.

Re: NanoLog – a nanosecond scale logging system for C++

#39

Post by gpderetta »

cma wrote: post #37
gpderetta wrote: post #25

Earlier quoted context omitted.

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously. That's why you are supposed to do your formatting in the background thread :). > doesn't matter how many millions of logs you can generate per second Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal…

Shared memory to a background process is better, in that it can survive most classes of process crash.

Yes it is significantly more robust, the downside is that you have to copy everything, including static vars and format strings. An option is to fork early on process startup and possibly share some memory.

We simply catch all signals and do a best effort attempt to flush the queue before aborting.

Re: NanoLog – a nanosecond scale logging system for C++

#40

Post by paulddraper »

Matthias247 wrote: post #33
gpderetta wrote: post #25

Earlier quoted context omitted.

> is that you will spend most of your time formatting strings if you do your file I/O asynchronously. That's why you are supposed to do your formatting in the background thread :). > doesn't matter how many millions of logs you can generate per second Usually the issue is not generating millions of line of logs (in whitch case dispatching to a background thread just adds overhead), but being able to log with minimal…

> That's why you are supposed to do your formatting in the background thread :). Which then has the drawback that every argument that gets transferred to the background thread must first be heap allocated or static - instead of only the final buffer being that. My gut feeling is that the additional allocation and memory management costs typically outweigh the cost of formatting in the current thread. Even if one pool…

Or you'd copy them.

Unless you do something dumb (like copy a vector when you only need to print the size), copying is significantly faster than string formatting.

Post reply on HN