Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoLog – a nanosecond scale logging system for C++

github.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: NanoLog – a nanosecond scale logging system for C++

#51

Post by profitnot »

muststopmyths wrote: post #3

I spent a lot of time trying to build a fast logging system in my last couple of jobs. The basic lesson (and I'm only talking about C/C++/C# here) is that you will spend most of your time formatting strings if you do your file I/O asynchronously. Since this system has a preprocessor mode, I assume they learnt the same lesson. The bigger lesson is that it really doesn't matter how many millions of logs you can generat…

This is, of course, true. Many fail with the mindset that if only they can build big enough haystacks, the needles will coalesce like shiny seaglass upon the shore, because surely they will hire the algorithmic wizards, and oh yes.. AI(!!) and ML(!!).

Still, this really could have useful applications in scientific applications where the goal is to capture extremely fast events in detail... super-colliders, fast-fusion events, etc...

Re: NanoLog – a nanosecond scale logging system for C++

#52

Post by paulddraper »

Matthias247 wrote: post #50
paulddraper wrote: post #40

Earlier quoted context omitted.

Or you'd copy them. Unless you do something dumb (like copy a vector when you only need to print the size), copying is significantly faster than string formatting.

Yes, but you need to have a location where to copy them to. They can’t be on the stack, since the logging task runs asynchronously. So each captured argument must be in some form heap allocated. Eg passing 2 strings as arguments and one integer might require 3 additional heap allocation - where the malloc/free overhead might outweigh the string formatting costs. You can try to optimize here with specialized allocator…

Good point

Re: NanoLog – a nanosecond scale logging system for C++

#53

Post by Mathnerd314 »

usefulcat wrote: post #28

The description says "the compilation builds a version of the NanoLog library that is non-portable, even between compilations of the same application". They also mention that a decompressor application also gets built and can be used to reconstitute a human-readable version of the log file. My question is, is the decompressor application also tightly coupled to a particular compilation? It sounds like the answer is y…

Did you skip over the section where they say to use the C++17 version that doesn't have this issue?

I looked in the paper to find the explanation for why the preprocessor version is non-portable; there wasn't anything clearly written but it seems to be related to how it assigns unique ID's to log statements.

Re: NanoLog – a nanosecond scale logging system for C++

#54

Post by cjensen »

muststopmyths wrote: post #3

I spent a lot of time trying to build a fast logging system in my last couple of jobs. The basic lesson (and I'm only talking about C/C++/C# here) is that you will spend most of your time formatting strings if you do your file I/O asynchronously. Since this system has a preprocessor mode, I assume they learnt the same lesson. The bigger lesson is that it really doesn't matter how many millions of logs you can generat…

Your points about logging with care saving more cycles that logging efficiently is very right.

That said, here's how I've handled efficient logging in realtime threads since the old days of 100MHz processors: postpone the formatting to a non-realtime thread. Assume your log_printf() will have no more than 10 int-sized args. Grab and save the 10 "ints". When doing the format in the non-realtime thread, pass in the 10 ints. Obviously this won't work with every CPU arch, but it does work fine for all the most common ones. Now every time you do a log_printf(), 1 pointer and 10 ints are copied to a buffer, which is very quick.

The catch is that your format string must be not be changed (e.g. use a string literal). Also any strings you want to print must also be const.

Re: NanoLog – a nanosecond scale logging system for C++

#56

Post by RafaGC »

Matthias247 wrote: post #50
paulddraper wrote: post #40

Earlier quoted context omitted.

Or you'd copy them. Unless you do something dumb (like copy a vector when you only need to print the size), copying is significantly faster than string formatting.

Yes, but you need to have a location where to copy them to. They can’t be on the stack, since the logging task runs asynchronously. So each captured argument must be in some form heap allocated. Eg passing 2 strings as arguments and one integer might require 3 additional heap allocation - where the malloc/free overhead might outweigh the string formatting costs. You can try to optimize here with specialized allocator…

On my logger the thread queue was(is) node based, so it was just a matter of making a bigger contiguous allocation and placing things there contiguously, the log entry and the data copies. 1 allocation.

Re: NanoLog – a nanosecond scale logging system for C++

#57

Post by RafaGC »

rafa1981 wrote: post #22

Author of mini-async-log and mini-async-log-c here. https://github.com/RafaGago/mini-async-log https://github.com/RafaGago/mini-async-log-c I wrote both some time ago and AFAIK they were the fastest, how does nanolog compare against them? At least before the source file preprocessor Nanolog was slower. Here I had a benchmark project with Nanolog on an old version, maybe you can PR an update: https://github.com/RafaGa…

As comments can't be edited: there are two nanolog projects on github. They may not be the same one.

Re: NanoLog – a nanosecond scale logging system for C++

#58

Post by gpderetta »

Matthias247 wrote: post #50
paulddraper wrote: post #40

Earlier quoted context omitted.

Or you'd copy them. Unless you do something dumb (like copy a vector when you only need to print the size), copying is significantly faster than string formatting.

Yes, but you need to have a location where to copy them to. They can’t be on the stack, since the logging task runs asynchronously. So each captured argument must be in some form heap allocated. Eg passing 2 strings as arguments and one integer might require 3 additional heap allocation - where the malloc/free overhead might outweigh the string formatting costs. You can try to optimize here with specialized allocator…

That would be a terrible implementation strategy. You would simply copy each argument into the communication ring buffer. You can also use specialised strategies for some types, for example string contents can be transmitted inline instead of copying the strings themselves.

Re: NanoLog – a nanosecond scale logging system for C++

#59

Post by [deleted] »

rafa1981 wrote: post #22

Author of mini-async-log and mini-async-log-c here. https://github.com/RafaGago/mini-async-log https://github.com/RafaGago/mini-async-log-c I wrote both some time ago and AFAIK they were the fastest, how does nanolog compare against them? At least before the source file preprocessor Nanolog was slower. Here I had a benchmark project with Nanolog on an old version, maybe you can PR an update: https://github.com/RafaGa…

[deleted]

Re: NanoLog – a nanosecond scale logging system for C++

#60

Post by pingyong »

userbinator wrote: post #43
MauranKilom wrote: post #9

Earlier quoted context omitted.

To the best of my knowledge, casting to char* is totally fine (inspecting an object as bytes) under certain constraints. What is not fine is pretending that an object lives at a position in memory where it does not (i.e. treating bytes of memory as some object through a reinterpret_cast away from char* ). Edit: To clarify, casting away from char* is of course allowed if you cast to whatever object type actually lives…

(i.e. treating bytes of memory as some object through a reinterpret_cast away from char ).* That's how people have used C and C++ for decades in low-level work, and it still works exactly as you'd expect, so stop saying "don't do it" because you're only encouraging the compiler-writer-UB-optimisation-nonsense crowd to make things even worse. It's already bad enough that they think the Holy Standard is the only thing…

Doesn't matter anymore, Clang and GCC will both make any and all optimizations they can find by assuming you will never hit UB, including just erasing all UB code. You can dislike it, but that's just the reality now, and ignoring it won't make it go away. The only way to change it would be a change in the standard.
Post reply on HN