Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compile-time JSON deserialization in C++

medium.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: Compile-time JSON deserialization in C++

#41

Post by OskarS »

chipdart wrote: post #33
stephc_int13 wrote: post #9

I am afraid of the compile-time cost. For this kind of things I tend to prefer using a simpler program (written in anything you like) to generate C or C++ instead of having the compile do the same thing much slowly. Meta programming can be good, but it is even better done with an actual meta program, IMO.

> I am afraid of the compile-time cost. Even though compilation time is the bane of C++, I think this concern regarding this specific usage is grossly overblown. I'm going to tell you why. With incremental builds you only rebuild whatever has changed in your project. Embedding JSON documents in a C++ app is the kind of thing that is rarely touched, specially if all your requirements are met by serializing docs at com…

There is another scenario where this is an issue: if this code ends up in a header which is included in a lot of places. You might say "that's dumb, don't do that", but there is a real tendency in C++ for things to migrate into headers (because they're templates, because you want them to be aggressively inlined, for convenience, whatever), and then headers get included into other headers, then without knowing it you suddenly have disastrous compile times.

Like, for this particular example, you might start out with a header that looks like:

    SomeData get_data_from_json(std::string_view json);
with nothing else in it, everything else in a .cpp file.

Then somebody comes around and says "we'd like to reuse the parsing logic to get SomeOtherData as well" and your nice, one-line header becomes

    template
    Ret get_data_from_json(std::string_view json) {
        // .. a gazillion lines of template-heavy code
    }
which ends up without someone noticing it in "CommonUtils.hpp", and now your compiler wants to curl up in a ball and cry every time you build.

It takes more discipline than you think across a team to prevent this from happening, mostly because a lot of people don't take "this takes too long to compile" as a serious complaint if it involves any kind of other trade-off.

Re: Compile-time JSON deserialization in C++

#42

Post by chipdart »

OskarS wrote: post #41
chipdart wrote: post #33

Earlier quoted context omitted.

> I am afraid of the compile-time cost. Even though compilation time is the bane of C++, I think this concern regarding this specific usage is grossly overblown. I'm going to tell you why. With incremental builds you only rebuild whatever has changed in your project. Embedding JSON documents in a C++ app is the kind of thing that is rarely touched, specially if all your requirements are met by serializing docs at com…

There is another scenario where this is an issue: if this code ends up in a header which is included in a lot of places. You might say "that's dumb, don't do that", but there is a real tendency in C++ for things to migrate into headers (because they're templates, because you want them to be aggressively inlined, for convenience, whatever), and then headers get included into other headers, then without knowing it you…

> There is another scenario where this is an issue: if this code ends up in a header which is included in a lot of places.

This is all on itself a sign that your project is fundamentally broken, but this is already covered by scenario b) incremental builds.

Even if for some reason you resist the urge of following best practices and not create your own problems, there are a myriad of techniques to limit the impact of touching a single file in your builds. Using a facade class to move your serialized JSON to an implementation detail of a class is perhaps the lowest effort one, but the textbook example would be something like a PIMPL.

The main problem with the build time of C++ projects are not the build times per se but clueless developers, who are oblivious to the problem domain, fumbling basic things and ending up creating their own problems. Once one of them stops to ask themself why is the project taking so much time to build, more often than not you find yourself a few commits away from dropping build times to a fraction of the cost. Even onboarding something like ccache requires no more than setting an environment variable.

Re: Compile-time JSON deserialization in C++

#43

Post by chipdart »

dctwin wrote: post #27
stephc_int13 wrote: post #9

I am afraid of the compile-time cost. For this kind of things I tend to prefer using a simpler program (written in anything you like) to generate C or C++ instead of having the compile do the same thing much slowly. Meta programming can be good, but it is even better done with an actual meta program, IMO.

Yes, I agree. I don't see much practical use in this. I was just surprised how (relatively) straightforwards this is to do, and thought it was more cool than useful

> Yes, I agree. I don't see much practical use in this.

Me too. The best example I can come up with is loading test data in automated tests, but even then I wouldn't use this sort of approach.

Re: Compile-time JSON deserialization in C++

#44

Post by chipdart »

threatripper wrote: post #11
adolph wrote: post #10

Earlier quoted context omitted.

Brings to mind the old story about a JSON DSL https://thedailywtf.com/articles/the-inner-json-effect

Is this real? It can't be real. Nobody can be this stupid. But then again it takes a special kind of person who doesn't understand satire to actually do something like that. Somebody, where they would say "we trained him wrong on purpose as a kind of a joke".

> Is this real? It can't be real. Nobody can be this stupid.

Having worked in an org with an official in-house genius who was terribly tight with a tech-illiterate leadership and faked his way into his status, I can't really tell. Throwing people under the bus, blaming the world around them for problems created by your brittle code, shunning best practices in favor of finger-pointing... This happens in small shops more often than we'd like believe.

As the saying goes, truth is stranger than fiction. Because fiction is expected to make sense.

Re: Compile-time JSON deserialization in C++

#45

Post by delfinom »

forrestthewoods wrote: post #8

I think the value of compile-time JSON deserialization is... well I was going to say zero but really it's negative. It's a cute trick, but please don't ever do this in a real project.

You clearly never wrote a Fizz Buzz enterprise grade application ;)

Re: Compile-time JSON deserialization in C++

#46

Post by actionfromafar »

chipdart wrote: post #42
OskarS wrote: post #41

Earlier quoted context omitted.

There is another scenario where this is an issue: if this code ends up in a header which is included in a lot of places. You might say "that's dumb, don't do that", but there is a real tendency in C++ for things to migrate into headers (because they're templates, because you want them to be aggressively inlined, for convenience, whatever), and then headers get included into other headers, then without knowing it you…

> There is another scenario where this is an issue: if this code ends up in a header which is included in a lot of places. This is all on itself a sign that your project is fundamentally broken, but this is already covered by scenario b) incremental builds. Even if for some reason you resist the urge of following best practices and not create your own problems, there are a myriad of techniques to limit the impact of…

Fundamentally broken, or waiting for modules to become a thing? I tried to use https://github.com/mjspncr/lzz3ᵃ for a few years but it became impractical to me to fiddle with tooling.

a: You don't have source file and header file, you put everything in one file and lzz sorts it out during build.

Re: Compile-time JSON deserialization in C++

#47

Post by sumtechguy »

bruce511 wrote: post #15
threatripper wrote: post #11

Earlier quoted context omitted.

Is this real? It can't be real. Nobody can be this stupid. But then again it takes a special kind of person who doesn't understand satire to actually do something like that. Somebody, where they would say "we trained him wrong on purpose as a kind of a joke".

I'm _pretty_ sure it's satire, but the fact that you and I can't say for sure is perhaps illustrative of the failure. I've encountered this pattern several times over my career. Some very smart programmer decides that for "reasons" the standard way to do something is "bad". (Usually "performance" or "bloat" are words bandied around.) They then happily architect a new system to replace the "old thing". Of course the n…

> There are reasons standard libraries exist

That right there. Before there is a standard lib for something if there are N people coding something up there could be N! ways to do something.

If you do not know about a standard lib or it doesn't exist there will be some wild code written.

It is when that standard library shows up you should at least consider just throwing your bespoke code away. Not always but should at least be considered. I personally have replaced thousands of lines of code and modules I wrote just by switching them to some existing library. The upside is if that standard lib does not do what I want I have enough knowledge to either bend it around so it does or I can fix it up (or put my bespoke code back). I know I am not that smart, but I know enough that my code is probably brittle and probably should be thrown away.

Also watch out for some 'standard libs'. Some of them are little more than someone's hobby project and have all the exact same issues you are trying to avoid. One project I worked on some guy had written a grid control. He was charging something like 10k a year to use it. But it was just one guy and I quote "i just touch it once or twice a year and drink margaritas on the beach". It was a bug prone riddled mess we spent a non insignificant amount of time fixing. We bought another one for a onetime fee of 500 bucks and it was wildly faster and more importantly had near zero bugs and a turn around time of 1-2 days if we found one.

Re: Compile-time JSON deserialization in C++

#48

Post by dctwin »

kookybakker wrote: post #35
dctwin wrote: post #28

Earlier quoted context omitted.

I was able to do the primitive long double result = 0.0; while (...) { if (json[head] == '.') ... result *= 10; result += json[head] - '0'; } in a constexpr function with no problem :)

How about floats in scientific notation?

You can do something similar, no? std::pow is not constexpr (most float stuff is not, presumably due to floating point state) but you can implement 10^x anyway

Re: Compile-time JSON deserialization in C++

#49

Post by dctwin »

silon42 wrote: post #38
dctwin wrote: post #27

Earlier quoted context omitted.

Yes, I agree. I don't see much practical use in this. I was just surprised how (relatively) straightforwards this is to do, and thought it was more cool than useful

Often I also find the opposite problem ... sure, you can do some stuff in (c++) metaprogramming, but can you (at compile time) generate a JSON/XML/YAML file that can be fed to some other part of the system?

The opposite 'toString' problem seems harder - I didn't try, but it should be possible now that std::string is constexpr.

I don't think you could parse it with, say, a class that has a std::string member (because of the transience restriction), but perhaps you can use lambdas that capture that string by reference, and call each other as appropriate?

As for exporting that as some sort of compiler artefact for use elsewhere, I am not sure how you would do that...

Re: Compile-time JSON deserialization in C++

#50

Post by pjc50 »

kazinator wrote: post #17
stephc_int13 wrote: post #9

I am afraid of the compile-time cost. For this kind of things I tend to prefer using a simpler program (written in anything you like) to generate C or C++ instead of having the compile do the same thing much slowly. Meta programming can be good, but it is even better done with an actual meta program, IMO.

> generate C or C++ instead of having the compile do the same thing much slowly That's a wild-assed guess. A JSON decoder right in the compiler could easily be faster than generation involving extra tool invocations and multiple passes. Also, if you use ten code generators for ten different features in a pipeline instead of ten compile-time things built into the language, will that still be faster? What if most files…

> You have to pass them through all the generators just in case; each generator decides whether the file contains anything that it knows how to expand.

The C# approach for this is that code generators operate as compiler plugins (and therefore also IDE plugins, so if you report an error from the code generator it goes with all the other compile errors). There is a two-pass approach where your plugin gets to scan the syntax tree quickly for "might be relevant" and then another go later; the first pass is cached.

A limitation of the plugin approach is that your codegen code itself has to be in a separate project that gets compiled first.

An argument in favor of separate-codegen is that if it breaks you can inspect the intermediate code, and indeed do things like breakpoints, logging and inspection in the code generator itself. The C++ approach seems like it might be hard to debug in some situations.

Post reply on HN