Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

blogs.embarcadero.com

Post reply on HN
Print view View original

151–160 of 187 posts

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#151

Post by fmxexpress »

ahepp wrote: post #3

>C++Builder with TwineCompile is a powerful productivity solution for multi-core machines compiling 1 million lines of code very quickly and can work better than the MAKE/GCC parallel compilation Jobs feature due to it’s deep IDE integration You're claiming this plugin has deeper IDE integration than `make`? I find that really, really difficult to believe. And if it's true, it seems like the solution is to either use…

Yes, it could be as simple as having Dev-C++ run a build every time a file is saved. Currently it does not do this. Remember, Dev-C++ didn't have -j support at all until I added it. TwineCompile does do this (background compile). Therefore the IDE is providing this functionality and has nothing really do to with make or the compiler.

TwineCompile is not a plugin wrapping the -j flag. It is a separate thing entirely unique to C++Builder. It does offer integration with MSBuild though.

The second part of that was the fall off. With the 1 million size files it only ever used half of the cores and each successive round of core compiles it would use even less cores. TwineCompile didn't seem to have that problem but this post was not about TwineCompile vs. MAKE -j so I did not investigate this farther.

I was expecting MAKE/GCC to blow me away and use all 64 cores full bore until complete and it did not do this.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#152

Post by fmxexpress »

com2kid wrote: post #26
simcop2387 wrote: post #16

Earlier quoted context omitted.

I believe this is one of the reasons for object libraries (or archives, the foo.a files on linux/unix), you can then link in all of the object files from one of those at link time without having to list them all at once. That won't get past the 2GB limit on executables but it will get past the command line length.

This is correct, a .lib file on Windows has a bunch of .obj files in it that you can then link together. You can also use command files[1] to pass options in instead of using the command line. [1] https://docs.microsoft.com/en-us/cpp/build/reference/linking... to pass

I searched for an option for make to do a command line file like this but didn't see one so struck out.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#153

Post by Wowfunhappy »

wtallis wrote: post #143
Wowfunhappy wrote: post #138

Earlier quoted context omitted.

I so happened to know this already, but I must say I've always found the approach somewhat weird. Wouldn't it make more sense to give the user all the available space, and then remove capacity slowly as blocks go bad? I guess they think people would be annoyed? Imagine if we treated batteries like SSDs, not allowing the use of a set amount of capacity so that it can be added back later, when the battery's "real" capa…

Shrinking the usable space on a block device is wildly impractical. The SSD has no awareness of how any specific LBA is being used, no way to communicate with the host system to find out what LBAs are safe to permanently remove. You can't just incrementally delete LBAs from the end of the drive, because important data gets stored there, like the backup GPT and OS recovery partitions. Filesystems also don't really lik…

That makes a lot of sense, thank you for the detailed answer!

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#154

Post by Too »

zajio1am wrote: post #57
klodolph wrote: post #17

Earlier quoted context omitted.

Make forces you to choose between being able to do full parallel builds or using recursive make, you can’t do both.

Why would you do recursive make? That is setup discouraged for decades ...

Because you included a third-party project into your build which is not compatible with your build system and you don't want to rewrite all their ninjafiles into makefiles to match.

Someone should really formalize a standard for declaring dependencies which build systems can share between each other.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#155

Post by KingOfCoders »

ska wrote: post #18
PragmaticPulp wrote: post #13

Fun experiment. The more pedestrian 5950X or the now bargain 3950X are great for anyone doing a lot of compiling. With the right motherboard they even have ECC RAM support. Game changer for workstations in the $1000–$2000 range. The more expensive Threadripper parts really shine when memory bandwidth becomes a bottleneck. In my experience, compiling code hasn’t been very memory bandwidth limited. However, some of my…

I suspect the biggest (build time) benefit to most c++ workflows and toolchains was the move to ubiquitous SSD. Prior to that in my experience excepting expensive RAID array dedicated build machines, it was really easy to build a system that would always be IO bound on builds. There of course were tricks to improve things but you still tended to hit that wall unless your CPUs were really under spec. edit: to be clear…

We had ultra fast HDDs as developers with sound proof housings because they were so loud. Glad for SSDs.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#156

Post by FeepingCreature »

mhh__ wrote: post #119
bserge wrote: post #48

Earlier quoted context omitted.

What can perform better?

I can build the D compiler (500k lines?) Warts and all in a second in my machine - and that's code that's not particularly complicated, but not at all optimized for compile times realistically.

To be fair, due to the fact that DMD can't depend on phobos it doesn't use a lot of the features that end up making D compiles slow, like lambdas and deep metaprogramming. DMD is in a way one of the least representative examples of a large D project.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#157

Post by klodolph »

Too wrote: post #154
zajio1am wrote: post #57

Earlier quoted context omitted.

Why would you do recursive make? That is setup discouraged for decades ...

Because you included a third-party project into your build which is not compatible with your build system and you don't want to rewrite all their ninjafiles into makefiles to match. Someone should really formalize a standard for declaring dependencies which build systems can share between each other.

A “standard for declaring dependencies” is, itself, a build system without the execution engine. The execution engine is not actually the hard part about making a build system. So, what you’re asking for is really just a standardized build system.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#158

Post by bjoli »

viktorcode wrote: post #56
bserge wrote: post #48

Earlier quoted context omitted.

What can perform better?

Other languages do perform better

TCC is fast enough that you can recompile the kernel every time you boot up.

I remember playing with TCC-boot back in 2015, and on a relatively beefy machine at the time I could compile the kernel in 5 seconds (or about 37mb/s iirc).

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#159

Post by kuschku »

adrian_b wrote: post #94
Sesse__ wrote: post #72

Earlier quoted context omitted.

Urm what? A modern NVMe drive will sustain ~2 GB/sec write. (See e.g. https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAXk6do4SSf8-128... , from Tom's Hardware)

Few SSDs can sustain such a speed for a long time. After they exhaust their temporary SLC cache they drop to much lower speeds. SSDs that have accumulated a large amount of writes might also make large pauses at random times during writing, for garbage collection. Good SSDs remain faster than HDDs even in these circumstances, but they are nevertheless much slower than when benchmarked for short times while they are n…

And that's why you buy 2-bit MLC, which can sustain these writes forever. Like e.g. the 970 Pro.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#160

Post by tomc1985 »

nickjj wrote: post #139
fctorial wrote: post #53

Earlier quoted context omitted.

Don't SSDs have a finite TBW? 50GB of writes everyday (possible on large projects) will consume that in a couple of months.

I've had a Crucial 256GB SSD (MX100) since early 2015 and I use it with Windows 10. WSL 2's file system is on there along with Docker, which I've been using full time since then. That means all of my source code, installing dependencies, building Docker images, etc. is done on the SSD. The SMART stats of the drive says it's at 88% health out of 100%, AKA it'll be dead when it reaches 0%. This is the wear and tear on…

As a counterpoint, I burn-tested several random M.2 NVME drives over a period of a month of 24/7 writes and reads and all but one model failed before the month was up
Post reply on HN