Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing architecture variants

discourse.ubuntu.com

Post reply on HN
Print view View original

121–130 of 152 posts

Re: Introducing architecture variants

#121

Post by JonChesterfield »

hajile wrote: post #118
ninkendo wrote: post #33

Earlier quoted context omitted.

> show that most packages show a slight (around 1%) performance improvement This takes me back to arguing with Gentoo users 20 years ago who insisted that compiling everything from source for their machine made everything faster. The consensus at the time was basically "theoretically, it's possible, but in practice, gcc isn't really doing much with the extra instructions anyway". Then there's stuff like glibc which h…

According to this[0] study of the Ubuntu 16.04 package repos, 89% of all x86 code was instructions were just 12 instructions (mov, add, call, lea, je, test, jmp, nop, cmp, jne, xor, and -- in that order). The extra issue here is that SIMD (the main optimization) simply sucks to use. Auto-vectorization has been mostly a pipe dream for decades now as the sufficiently-smart compiler simply hasn't materialized yet (and m…

The sufficiently smart vectoriser has been here for decades. Cuda is one. Uses all the vector units just fine, may struggle to use the scalar units.

Re: Introducing architecture variants

#122

Post by dima55 »

julian-klode wrote: post #66
amelius wrote: post #31

Can we please have an "apt rollback" function?

Yes sure apt (3.1.7) unstable; urgency=medium . [ Julian Andres Klode ] * test-history: Adjust for as-installed testing . [ Simon Johnsson ] * Add history undo, redo, and rollback features

Exciting. I just looked for docs about these new features, and can't find anything. Can you point us to these? Thanks!

Re: Introducing architecture variants

#123

Post by gpm »

tclancy wrote: post #109
gpm wrote: post #47

Earlier quoted context omitted.

What a strange comparison. If you're negotiating deals worth billions of dollars, or even just millions, I'd strongly suggest not doing so with a hangover.

Sounds like someone never read Sun Tzu. (Not really, I just know somewhere out there is a LinkedInLunatic who has a Business Philosophy based on being hungover.)

Appear drunk when you are sober, and sober when you are drunk

- Sun Zoo

Re: Introducing architecture variants

#124

Post by kccqzy »

hajile wrote: post #118
ninkendo wrote: post #33

Earlier quoted context omitted.

> show that most packages show a slight (around 1%) performance improvement This takes me back to arguing with Gentoo users 20 years ago who insisted that compiling everything from source for their machine made everything faster. The consensus at the time was basically "theoretically, it's possible, but in practice, gcc isn't really doing much with the extra instructions anyway". Then there's stuff like glibc which h…

According to this[0] study of the Ubuntu 16.04 package repos, 89% of all x86 code was instructions were just 12 instructions (mov, add, call, lea, je, test, jmp, nop, cmp, jne, xor, and -- in that order). The extra issue here is that SIMD (the main optimization) simply sucks to use. Auto-vectorization has been mostly a pipe dream for decades now as the sufficiently-smart compiler simply hasn't materialized yet (and m…

The highway library is exactly the kind of a simpler option to use SIMD. Less efficient than hand written assembler but you can easily write good enough SIMD for multiple different architectures.

Re: Introducing architecture variants

#125

Post by physicsguy »

colechristensen wrote: post #23
physicsguy wrote: post #11

This is quite good news but it’s worth remembering that it’s a rare piece of software in the modern scientific/numerical world that can be compiled against the versions in distro package managers, as versions can significantly lag upstream months after release. If you’re doing that sort of work, you also shouldn’t use pre-compiled PyPi packages for the same reason - you leave a ton of performance on the table by not…

Most of the scientific numerical code I ever used had been in use for decades and would compile on a unix variant released in 1992, much less the distribution version of dependencies that were a year or two behind upstream.

I mean that’s just lucky and totally depends on your field and what is normal - just as an example, we used the LLNL SUNDIALS package for implicit time integration. On Ubuntu 24.04 the latest version is 6.4.1 where the latest published is v7.5.0. We found their major version releases tended to require changes.

There’s also the difference between being able to run and being able to run optimised. At least 5 years ago, the Ubuntu/Debian builds of FFTW didn’t include the parallelised OpenMP library.

In a past life I did HPC support and I recommend the Spack package manager a lot to people working in this area because you can get optimised builds with whatever compiler tool chain and options you need quite easily that way.

Re: Introducing architecture variants

#126

Post by physicsguy »

niwtsol wrote: post #25
physicsguy wrote: post #11

This is quite good news but it’s worth remembering that it’s a rare piece of software in the modern scientific/numerical world that can be compiled against the versions in distro package managers, as versions can significantly lag upstream months after release. If you’re doing that sort of work, you also shouldn’t use pre-compiled PyPi packages for the same reason - you leave a ton of performance on the table by not…

Thanks for sharing this. I'd love to learn more about micro-architectures and instruction sets - would you have any recommendations for books or sources that would be a good starting place?

My experience is mostly practical really - the trick is to learn how to compile stuff yourself.

If you do a typical: "cmake . && make install" then you will often miss compiler optimisations. There's no standard across different packages so you often have to dig into internals of the build system and look at the options provided and experiment.

Typically if you compile a C/C++/Fortran .cpp/.c/.fXX file by hand, you have to supply arguments to instruct the use of specific instruction sets. -march=native typically means "compile this binary to run with the maximum set of SIMD instrucitons that my current machine supports" but you can get quite granular doing things like "-march=sse4,avx,avx2" for either compatibility reasons or to try out subsets.

Re: Introducing architecture variants

#127

Post by mwhudson »

zozbot234 wrote: post #16

What are the changes to dpkg and apt? Are they being shared with Debian? Could this be used to address the pesky armel vs. armel+hardfloat vs. armhf issue, or for that matter, the issue of i486 vs. i586 vs. i686 vs. the many varieties of MMX and SSE extensions for 32-bit? (There is some older text in the Debian Wiki https://wiki.debian.org/ArchitectureVariants but it's not clear if it's directly related to this effor…

> Could this be used to address the pesky armel vs. armel+hardfloat vs. armhf issue

No, because those are different ABIs (and a debian architecture is really an ABI)

> the issue of i486 vs. i586 vs. i686 vs. the many varieties of MMX and SSE extensions for 32-bit?

It could be used for this but it's about 15 years too late to care surely?

> (There is some older text in the Debian Wiki https://wiki.debian.org/ArchitectureVariants but it's not clear if it's directly related to this effort)

Yeah that is a previous version of the same design. I need to get back to talking to Debian folks about this.

Re: Introducing architecture variants

#128

Post by mwhudson »

dfc wrote: post #17

> you will not be able to transfer your hard-drive/SSD to an older machine that does not support x86-64-v3. Usually, we try to ensure that moving drives between systems like this would work. For 26.04 LTS, we’ll be working on making this experience cleaner, and hopefully provide a method of recovering a system that is in this state. Does anyone know what the plans are to accomplish this?

I am probably going to be the one implementing this and I don't know what I am going to do yet! At the very least we need the failure mode to be better (currently you get an OOPS when the init from the initrd dies due to an illegal instruction exception)

Re: Introducing architecture variants

#129

Post by mwhudson »

stabbles wrote: post #24

Seems like this is not using glibc's hwcaps (where shared libraries were located in microarch specific subdirs). To me hwcaps feels like a very unfortunate feature creep of glibc now. I don't see why it was ever added, given that it's hard to compile only shared libraries for a specific microarch, and it does not benefit executables. Distros seem to avoid it. All it does is causing unnecessary stat calls when running…

No it's not using hwcaps. That would only allow optimization of code in shared libraries, would be irritating to implement in a way that didn't require touching each package that includes shared libraries and would (depending on details) waste a bunch of space on every users system. I think hwcaps would only make sense for a small number of shared libraries if at all, not a system wide thing.
Post reply on HN