311–320 of 375 posts
Isn't GeoJSON a really bad tool for what you are trying to do ? Just use GeoParquet: Converter: https://geoparquet.org/convert/ It'll scan way less data and be 90% faster than your 90% faster jq.
"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…
Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…
This is a common pain point. Write in a language that makes sense for the project. Then people tell you that you should have used this other language, for reasons. Use a compression algo that makes sense for your data. Then people will tell you why you are stupid and should have used this other algo. My most recent memory of this was needing to compress specific long json strings to fit in Dynamo. I exhaustively test…
If you decide to go down this road just use Gentoo.
I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
Earlier quoted context omitted.
Everything outperforms glibc malloc. It's essentially malpractice that distros continue to use it instead of mimalloc or jemalloc.
It's been awhile since I looked into this, but it's not necessarily an easy change. glibc malloc has debugging APIs; a distro can't easily replace it without either emulating the API or patching programs that use it.
From a end developer perspective: I have no particular familiarity with mimalloc, but I know jemalloc has pretty extensive debugging functionality (not API compatible with glibc malloc of course).
Reading this, I wonder a few things that seem distro quick wins: 1) Why don't distros replace the glibc allocator with one of the better ones? 2) Why don't distros allow for making server-specific builds for only a few packages? You don't have to pay the compilation cost for everything, just a list of 2 or 3 packages.
Earlier quoted context omitted.
Portage on top of Fuchsia, probably. There's not all that much more to Gentoo all things considered.
Zircon is miles apart from linux. It’s like saying you’d run Portage on top of Windows. You and what army?
Earlier quoted context omitted.
ASLR is technically a form of security by obscurity. The obscurity here being the memory layout. The reason nobody treated it that way was the high entropy that ASLR had on 64-bit, but the ASLR⊕Cache attack has undermined that significantly. You really do not want ASLR to be what determines whether an attacker takes control of your machine if you care about having a secure system.
You are confusing randomization, a legitimate security mechanism, with security by obscurity. ASLR is not security by obscurity. Please spend the time on understanding the terminology rather than regurgitating buzz words.
https://web.archive.org/web/20240123122515if_/https://www.sy...
Earlier quoted context omitted.
While I do agree with the general sentiment, I think the default should be to use the safer ones that can handle multi threaded usage. If someone wants to use a bad allocator like glibc that doesn't handle concurrency well, then they should certainly be free to switch.
So now you’re using std vector and it’s taking a lock for no reason, even though push_back on the same vector across threads isn’t thread safe to begin with. Haphazard multithreading is not a sane default. I understand a million decisions have been made so that we can’t go flip that switch back off, but we’ve got to learn these lessons for the future.
Even glibc claims to be multithreading safe even if it tends to not return or reuse all freed memory.
Earlier quoted context omitted.
ASLR is technically a form of security by obscurity. The obscurity here being the memory layout. The reason nobody treated it that way was the high entropy that ASLR had on 64-bit, but the ASLR⊕Cache attack has undermined that significantly. You really do not want ASLR to be what determines whether an attacker takes control of your machine if you care about having a secure system.
The defining characteristic of security through obscurity is that the effectiveness of the security measure depends on the attacker not knowing about the measure at all. That description doesn’t apply to ASLR.
https://web.archive.org/web/20240123122515if_/https://www.sy...
