Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

blogs.embarcadero.com

Post reply on HN
Print view View original

91–100 of 187 posts

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#91

Post by Dylan16807 »

fctorial wrote: post #59
EricE wrote: post #42

Earlier quoted context omitted.

A way to have your cake and eat it too - check out Primocache. It's pretty inexpensive disk caching software (especially for Windows Server which is where I really leverage it!). Pair it with an Optane for L2 cache and it will speed up normal SSD use too ;)

How does it compare to default linux caching algorithm?

The default one that caches disk data in memory? It solves a very different problem by caching data from slower disks onto faster disks. It can accelerate reads and it can also act as a writeback cache.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#92

Post by klodolph »

zajio1am wrote: post #57
klodolph wrote: post #17

Earlier quoted context omitted.

Make forces you to choose between being able to do full parallel builds or using recursive make, you can’t do both.

Why would you do recursive make? That is setup discouraged for decades ...

Because it can be hard to maintain non-recursive make systems for large projects. Just because recursive make is discouraged does not mean that the alternative is without its own drawbacks.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#93

Post by titzer »

bserge wrote: post #48
titzer wrote: post #27

Earlier quoted context omitted.

17,300 lines/sec per core. That's embarrassingly slow IMHO.

What can perform better?

Lots of single-pass compilers can achieve 1MLOC/s. But the main problem is that C++ has an O(n^2) compilation model due to header explosion. Also, expanding C++ templates is very computationally intensive.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#94

Post by adrian_b »

Sesse__ wrote: post #72
trhway wrote: post #61

Earlier quoted context omitted.

SSD have hard time sustaining 200-400Mb/s write where is 4 HDD do is easily. Our case isn't that much about IOPS. Anyway, reasonably available SSDs have up to [1000 x SSD size] total write limit, so doing couple of 400G builds/day would use up the 1TB drive in 3 years. At worst times we had to develop&maintain 5 releases in parallel instead of regular 2-3.

Urm what? A modern NVMe drive will sustain ~2 GB/sec write. (See e.g. https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAXk6do4SSf8-128... , from Tom's Hardware)

Few SSDs can sustain such a speed for a long time. After they exhaust their temporary SLC cache they drop to much lower speeds. SSDs that have accumulated a large amount of writes might also make large pauses at random times during writing, for garbage collection. Good SSDs remain faster than HDDs even in these circumstances, but they are nevertheless much slower than when benchmarked for short times while they are new.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#95

Post by drmpeg »

bullen wrote: post #47
jcelerier wrote: post #43

Earlier quoted context omitted.

You just need more ram. I 'ever compile at less than -j$(ncpu). Hard with less than 32 GB tho - a single clang instance can easily eat upwards of 1gb of ram

Aha, I only compile on ARM so I got no room to increase RAM... Is it the same with g++? I have 4GB so I should be able to compile with 4 cores, but the processes only fill 2-3 cores even when I try make -j8 on a 8 core machine and then locks the entire OS until it craps out?! Something is fishy...

A swap file is your friend.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#96

Post by dvfjsdhgfv »

cosmotic wrote: post #11
coliveira wrote: post #7

It is a good thing that Embarcadero is keeping alive this technology to create desktop apps from the early 2000s that was abandoned by MS and other large companies in favor of complex Web-based apps.

Someone has to build the native electron wrapper

If you mean wrapping native widgets, this wouldn't solve much - you would still need some language to take care of the logic, like a JavaScript engine. At this point just using Electron is simply easier for devs, and as much as we hate it, realistically speaking it's still better than nothing.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#97

Post by nwmcsween »

trhway wrote: post #61
Retric wrote: post #51

Earlier quoted context omitted.

If your SSD would be at 100% utilization it’s going to take a lot of HDD to reach that kind of bandwidth. To the point where for high bandwidth loads SSD’s actually cost less even if you have to replace them regularly. 100% utilization and 30x the bandwidth = 30x as many HDD. Alternatively, if HDD’s are an option you’re a long way from 100% utilization.

SSD have hard time sustaining 200-400Mb/s write where is 4 HDD do is easily. Our case isn't that much about IOPS. Anyway, reasonably available SSDs have up to [1000 x SSD size] total write limit, so doing couple of 400G builds/day would use up the 1TB drive in 3 years. At worst times we had to develop&maintain 5 releases in parallel instead of regular 2-3.

4 HDDs can do 200-400MB/s _sequential_ IO, 1 modern SSD can do 150-200MB/s _random_ IO and 400MB/s sequential IO while 4 HDDs would have a hard time doing IIRC more than 8MB/s random IO

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#98

Post by bmurphy1976 »

55873445216111 wrote: post #79
bmurphy1976 wrote: post #65

Earlier quoted context omitted.

Where can you get decent ECC ram for a reasonable price? I was on the hunt recently for ECC RAM for my new desktop and I gave up and pulled the trigger on low latency non-ECC RAM. Availability seems to be pretty terrible at the moment.

You can get ECC UDIMMs from Supermicro. They are rebranded Micron DIMMs. ECC memory is not going to go as high of frequencies as you might be looking for. They will only go up to the officially validated speed of the CPUs. https://store.supermicro.com/16gb-ddr4-mem-dr416l-cv02-eu26....

Oh thanks, I didn't even think to check Supermicro! The prices are reasonable as well.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#99

Post by officeplant »

dragontamer wrote: post #39
leetcrew wrote: post #37

Earlier quoted context omitted.

a build server recompiling multiple branches over and over in response to changes.

And logging all of those unit tests associated with all of those builds (and rolling over those logs with TRACE level debugging). Every build gets fully tested at maximum TRACE logging, so that anyone who looks at the build / test later can search the logs for the bug. 8TBs of storage is a $200 hard drive. Fill it up, save everything. Buy 5 hard drives, copy data across them redundantly with ZFS and stuff. 1TBs of SS…

My mind was blown when I had to send a fully logged 5 minutes of system operation to a friend for diagnostics (MacOS 11 on an M1 Mini). He wasn't joking when he said don't go over a few minutes because the main 256GB system drive almost ran out of space in that time. After getting it compressed down from 80GB and sent over I got my mind blown again when he explained he had to move to his workstation with 512+gb of ram just to open the damn file.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#100

Post by nine_k »

dekhn wrote: post #74
fctorial wrote: post #53

Earlier quoted context omitted.

Don't SSDs have a finite TBW? 50GB of writes everyday (possible on large projects) will consume that in a couple of months.

I've swapped hundreds of terabytes to a terabyte SSD (off the shelf cheapie) with no recognizable problems (the gigapixel panoramas look fine).

SSDs avoid catastrophic write failures by retiring damaged blocks. Check the reported capacity; it may have shrunk :) Before you ever see bad blocks, the drive will expend spare blocks; this means that a new drive you buy has 10% more capacity than advertised, and this capacity will be spent to replace blocks worn by writes.
Post reply on HN