Earlier quoted context omitted.
That was me that filed the Itanium test suite failure. :)
The Itanic was kind of great :). I'm convinced it helped sink SGI.
SGI and HP! Intel should have a statue of Rick Belluzzo on they’r campus.
Jemalloc Postmortem
141–150 of 250 posts
Re: Jemalloc Postmortem
#142Oh that's interesting. jemalloc is the memory allocator used by redis, among other projects. Wonder what the performance impact will be if they have to change allocators.
Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.
Some people believe everything must always be constantly tweaked, redone, broken and fixed, and churned for no reason. The only things that need to be fixed in mature, working software are bugs and security issues. It doesn't magically stop working or get "stale" unless dependencies, the OS, or build tools break.
Re: Jemalloc Postmortem
#143Looking at all the comments and lightly browsing the source code, I'm amazed. Both at how much impact a memory allocator can make, but also how much code is involved. I'm not really sure what I expected, but somehow I expect a memory allocator to be ... smaller, simpler perhaps?
Memory allocators can be simple. In fact it was an assignment for a course in the 2nd year of my CS degree to make an (almost) complete allocator.
However it is typically always more complex to make production quality software, especially in a performance sensitive domain.
Re: Jemalloc Postmortem
#144Lesson: Don't let one megacorp dominate or take over your FOSS project. Push back somewhat and say "no" to too much help from one source.
Re: Jemalloc Postmortem
#145Oh that's interesting. jemalloc is the memory allocator used by redis, among other projects. Wonder what the performance impact will be if they have to change allocators.
Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.
> Sometimes software development is largely "done"
Lol absolutely not
Re: Jemalloc Postmortem
#146Looking at all the comments and lightly browsing the source code, I'm amazed. Both at how much impact a memory allocator can make, but also how much code is involved. I'm not really sure what I expected, but somehow I expect a memory allocator to be ... smaller, simpler perhaps?
Memory allocators can be simple. In fact it was an assignment for a course in the 2nd year of my CS degree to make an (almost) complete allocator. However it is typically always more complex to make production quality software, especially in a performance sensitive domain.
Naive allocators are very easy: just subdivide RAM and defragment only when absolutely necessary (if virtual memory is unavailable). Performant allocators are hard.
I think we lost a great deal of potential when ORCA was too tied to Pony and not extracted to a framework, tool, and/or library useful outside of it such as integrated or working with LLVM.
Re: Jemalloc Postmortem
#147I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…
> we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file
Why not? I mean this is complete drive-by comment, so please correct me, but there was a fully staffed team at Meta that maintained it, but was not in the best place to manage the issues?
Re: Jemalloc Postmortem
#148[dead]
Re: Jemalloc Postmortem
#149Earlier quoted context omitted.
They are compatible but that's not the point. If it were included it would instantly become a LGPL hard-fork because of any subsequently added line of code, if not by "virality" of the glibc license, at least because any glibc author code addition would be LGPL, per GNU project policy/ideology. Also also this would he a hard bar to pass: https://sourceware.org/glibc/wiki/CopyrightFSForDisclaim As I recall this is wha…
What prevents apple from working with gpl-style licenses is strict hatred towards code that they can't use without opensourcing it. So this is what prevents them from contributing to gpl projects: the need to control access to code. Llvm is OK for them from this point of view: upstream is open but they can maintain and distribute their proprietary fork.
> What prevents apple from working with gpl-style licenses is strict hatred towards code that they can't use without opensourcing it.
Specifically regarding the C blocks feature introduced in Snow Leopard, as I recall, Apple wrote implementations for both clang and gcc, attempted to upstream the gcc patchset, said gcc patchset was obviously under a GPL license, but the GCC team threw a fit because it wanted the code copyright to be attributed to the FSF, and that ended up as a stalemate.
If there was any hatred they could literally have skipped the whole gcc implementation + patchset upstreaming attempt altogether. Also they did have patchsets of various sizes on other projects, whose code ends up obviously being GPL as well.
The "hatred" came later with the GPLv3 family and the patent clause, which is a legal landmine, the FSF stating that signing apps is incompatible with the GPLv3, and getting hung up on copyright transfer.
From https://lwn.net/Articles/405417/
> Apple's motives may not be pure, but it has published the code under the license required and it's the FSF's own copyright assignment policies that block the inclusion. The code is available and licensed appropriately for the version of GCC that Apple adopted. It might be nice if Apple took the further step of assigning copyright to the FSF, but the GPLv3 was not part of the bargain that Apple agreed to when it first started contributing to GCC.
The intent behind such copyright transfer is generally so that the recipient of the transfer can relicense without having to ask all contributors. Essentially as a contributor agreeing to a transfer means ceding up control on the license that you initially contributed under.
Read another way:
- the FSF says "this code is GPLv2"
- someone contributes under that GPLv2 promise, cedes copyright to the FSF because it's "the process"
- the FSF says "this code is now GPLv3 exclusively"
- that someone says "but that was not the deal!"
- the FSF says "I am altering the deal, pray I don't alter it any further."
Re: Jemalloc Postmortem
#150Earlier quoted context omitted.
> Semi-related: one thing that most people never think about: it is exactly the same amount of work for the kernel to zero a page of memory (in preparation for a future mmap) as for a userland process to zero it out (for its own internal reuse) Possibly more work since the kernel can't use SIMD
Why is that? Doesn't Linux use SIMD for the crypto operations?
Allowing SIMD instructions to be used arbitrarily in kernel actually has a fair penalty to it. I'm not sure what Linux does specifically, but:
When a syscall is made, the kernel has to backup the user mode state of the thread, so it can restore it later.
If any kernel code could use SIMD registers, you'll have to backup and restore that too, and those registers get big. You could easily be looking at adding a 1kb copy to every syscall, and most of the time it wouldn't be needed.