Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

161–170 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#161

Post by silisili »

smallstepforman wrote: post #127

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…

This is a common pain point.

Write in a language that makes sense for the project. Then people tell you that you should have used this other language, for reasons.

Use a compression algo that makes sense for your data. Then people will tell you why you are stupid and should have used this other algo.

My most recent memory of this was needing to compress specific long json strings to fit in Dynamo. I exhaustively tested every popular algo, and Brotli came out far ahead. But that didn't stop every passerby from telling me that zlib is better.

It's rather exhausting at times...

Re: Make Ubuntu packages 90% faster by rebuilding them

#162

Post by eru »

ryao wrote: post #80
eru wrote: post #57

Earlier quoted context omitted.

I used Gentoo for a while, but the temptation to endlessly fiddle with everything always let me to eventually break the system. (It's not Gentoo's fault, it's mine.) Afterwards I moved to ArchLinux, and that has been mostly fine for me. If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?

Gentoo lets you do all of the tweaks mentioned here within the system package manager, so you still get security updates for your tweaked build. You can also install Gentoo on top of another system via Gentoo Prefix for use as a userland packages manager: https://wiki.gentoo.org/wiki/Project:Prefix

> Gentoo lets you do all of the tweaks mentioned here within the system package manager, so you still get security updates for your tweaked build.

Yes, Gentoo is great. I'm just saying that for me it was too much of a temptation.

Re: Make Ubuntu packages 90% faster by rebuilding them

#163

Post by eru »

serbuvlad wrote: post #130
eru wrote: post #57

Earlier quoted context omitted.

I used Gentoo for a while, but the temptation to endlessly fiddle with everything always let me to eventually break the system. (It's not Gentoo's fault, it's mine.) Afterwards I moved to ArchLinux, and that has been mostly fine for me. If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?

I'd like to bring attention to the ALHP repos[1]. These are the Arch packages built for x86-64-v2, x86-64-v3 and x86-64-v4, which are basically names for different sets of x86-64 extensions. Selecting the highest level supported by your processor should get you most of the way to -march=native, without the hassle of compiling it yourself. It also enables -O3 and LTO for all packages. [1]: https://github.com/an0nfunc/…

Nice, I'll try them out!

LTO is great, but I have my doubts about -O3 (vs the more conservative -O2).

UPDATE: bah, ALHP repos don't support the nvidia drivers. And I don't want to muck around with setting everything up again.

Another update: I moved to nvidia-open, so now I can try the suggested repos.

Re: Make Ubuntu packages 90% faster by rebuilding them

#166

Post by vasco »

teitoklien wrote: post #40

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…

Never seen the HN version of the 'install gentoo' meme before, more sophisticated definitely.

> The goal of Gentoo is to have an operating system that builds all programs from source, instead of having pre-built binary packages. While this does allow for advanced speed and customizability, it means that even the most basic components such as the kernel must be compiled from source. It is known through out the Linux community as being a very complex operating system because of its daunting install process. The default Gentoo install boots straight to a command prompt, from which the user must manually partition the disk, download a package known as a "Stage 3 tarball", extract it, and build the system up by manually installing packages. New or inexperienced users will often not know what to do when they boot in to the installer to find there is no graphical display. Members of /g/ will often exaggerate the values of Gentoo, trying to trick new users in to attempting to install it.

Re: Make Ubuntu packages 90% faster by rebuilding them

#167

Post by dgan »

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes. i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

Re: Make Ubuntu packages 90% faster by rebuilding them

#168

Post by nurettin »

dgan wrote: post #167

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes . i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

Does ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?

Re: Make Ubuntu packages 90% faster by rebuilding them

#169

Post by Hackbraten »

dgan wrote: post #167

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes . i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

There might me some kind of malware scanner at play.

Maybe it's time to rebuild those with mimalloc, too.

Re: Make Ubuntu packages 90% faster by rebuilding them

#170

Post by dgan »

nurettin wrote: post #168
dgan wrote: post #167

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes . i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

Does ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?

they don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content
Post reply on HN