Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

blogs.embarcadero.com

Post reply on HN
Print view View original

121–130 of 187 posts

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#121

Post by pdimitar »

1996 wrote: post #102
pdimitar wrote: post #62

Earlier quoted context omitted.

I recently bought an external 32GB SLC SSD (in the form factor of an USB pendrive). Its random read/write speeds are quite insane (130+ MB/s both) while consumer SSDs like the Samsung 850 Evo barely manage 30 MB/s read/write. It's also advertised as very durable. I plan on using a couple of those as ZFS metadata and small block caches for my home NAS, we'll see how it goes but people generally and universally praise…

> I recently bought an external 32GB SLC SSD (in the form factor of an USB pendrive). Its random read/write speeds are quite insane (130+ MB/s both) Which brand/spec?

https://www.aliexpress.com/item/32846479966.html?spm=a2g0s.9...

Vtran / eVtran. They have favourable reviews, too.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#122

Post by Retric »

adrian_b wrote: post #94
Sesse__ wrote: post #72

Earlier quoted context omitted.

Urm what? A modern NVMe drive will sustain ~2 GB/sec write. (See e.g. https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAXk6do4SSf8-128... , from Tom's Hardware)

Few SSDs can sustain such a speed for a long time. After they exhaust their temporary SLC cache they drop to much lower speeds. SSDs that have accumulated a large amount of writes might also make large pauses at random times during writing, for garbage collection. Good SSDs remain faster than HDDs even in these circumstances, but they are nevertheless much slower than when benchmarked for short times while they are n…

You can get SSDs without such issues. At least for the kind of long sequential writes followed by deletes where HDD are are even vaguely comparable, good SSD’s can maintain very close to their theoretical maximum speeds indefinitely.

They only really slow down in usage patterns that trash traditional HDD’s, which have much worse disk fragmentation issues.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#123

Post by steveklabnik »

moonchild wrote: post #116
josephg wrote: post #110

Earlier quoted context omitted.

I wonder if it’d be possible / useful to implement a rust or C compiler that worked that way. Basically instant startup time after a code change is a delightful feature.

I've been bandying around ideas lately for an extremely fast c compiler that, among other things, caches machine code for individual function declarations. You wouldn't realistically be able to get to the level of speed as js, though, because js is a dynamic language. With a statically typed language like c or rust, any time you change a declaration, you have to re-typecheck everything that depends on that declaratio…

Strictly speaking, Rust doesn't have templates, or "template expansion", but there is a thing called "trait resolution" which does take effort to solve, that is related but not the same thing as what C++ calls "template expansion."

If you'd like to learn more the current way this works is described here: https://rustc-dev-guide.rust-lang.org/traits/resolution.html

As it mentions, this is being re-written, and eventually, will use an SLG solver https://rustc-dev-guide.rust-lang.org/traits/chalk.html

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#124

Post by zlynx »

dekhn wrote: post #111
nine_k wrote: post #100

Earlier quoted context omitted.

SSDs avoid catastrophic write failures by retiring damaged blocks. Check the reported capacity; it may have shrunk :) Before you ever see bad blocks, the drive will expend spare blocks; this means that a new drive you buy has 10% more capacity than advertised, and this capacity will be spent to replace blocks worn by writes.

Now I'm curious how the kernel responds to having swap partitions resized under it.

Not quite how it works. The SSD will never have a capacity lower than its rated capacity.

For the failures I've seen, once the SSD goes to do a write operation and there's no free blocks left, it will lock into read-only mode. And at that point it is dead. Time to get a new one.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#125

Post by zlynx »

fctorial wrote: post #58
trhway wrote: post #24

Earlier quoted context omitted.

i (and some teamates) actually put HDDs on some workstations as SSD just die after 2-3 years of active build on them and with modern HDDs you have practically unlimited storage while you can have only limited number of 400G builds on SSD (the org has psychological barriers to having more than 1-2Tb SSD in a machine) and the SSD start to have perf issues when at 70-80% capacity . With HDD the build time didn't change…

What are the signs of an SSD that's about to die?

SMART monitoring.

In my experience the SMART data on SSDs is pretty good at pointing to failures.

Either that or the SSD simply disappears from the system. But that happens to HDDs also.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#126

Post by throwaway81523 »

How many times are they going to repeat the search phrases like "one billion lines"? It's reached the point where SEO obstructs human readability. It was cool that Object Pascal (maybe a descendant of Turbo Pascal) compiled 1e9 lines of Pascal in 5 minutes on the 64 core box. Scrolling way through the article, it looks like they had enough trouble setting up their parallel Windows C++ build environment on 64 cores that they ended up running 4 instances on 16 cores each, and splitting the source files among the instances. The build then took about 15 minutes on 64 cores, which is faster than I'd have expected.

This all seems kind of pointless since distributed C++ compilation has been a thing for decades, so they could have used a cluster of Ryzens instead of "zowie look at our huge expensive single box".

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#127

Post by blt »

formerly_proven wrote: post #4

[Not "real" C++ code, benchmark is for compiling 14492754 copies of a fairly simple C function]

Yeah, I would say this title is a little misleading. The example doesn't use any of the C++ features that cause long compile times, like templates and the STL.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#129

Post by wtallis »

nine_k wrote: post #100
dekhn wrote: post #74

Earlier quoted context omitted.

I've swapped hundreds of terabytes to a terabyte SSD (off the shelf cheapie) with no recognizable problems (the gigapixel panoramas look fine).

SSDs avoid catastrophic write failures by retiring damaged blocks. Check the reported capacity; it may have shrunk :) Before you ever see bad blocks, the drive will expend spare blocks; this means that a new drive you buy has 10% more capacity than advertised, and this capacity will be spent to replace blocks worn by writes.

SSDs never shrink the reported capacity in response to defective/worn-out blocks. Instead, they have spare area that is not accessible to the user. The SMART data reported by the drive includes an indicator of how much spare area has been used and how much remains. When the available spare area starts dropping rapidly, you're approaching the drive's end of life.

Re: Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64 Cores

#130

Post by pirocks »

bullen wrote: post #47
jcelerier wrote: post #43

Earlier quoted context omitted.

You just need more ram. I 'ever compile at less than -j$(ncpu). Hard with less than 32 GB tho - a single clang instance can easily eat upwards of 1gb of ram

Aha, I only compile on ARM so I got no room to increase RAM... Is it the same with g++? I have 4GB so I should be able to compile with 4 cores, but the processes only fill 2-3 cores even when I try make -j8 on a 8 core machine and then locks the entire OS until it craps out?! Something is fishy...

This is just a low end/uncommon hardware problem. I typically do make -j16 on a 4 core x86 system and it just works. You are probably running out of ram and the swapping resulting in that instability.
Post reply on HN