brilliant except for weird edge case failure modes that aren’t obvious, like random dns lookup failures or random network dropouts.
Alpine Linux: Brilliant Linux Distro
81–90 of 98 posts
Re: Alpine Linux: Brilliant Linux Distro
#82I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues. eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...
Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while. AWS support's only advice was "don't use Alpine"; annoyingly, switching the container…
Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.
Re: Alpine Linux: Brilliant Linux Distro
#83Earlier quoted context omitted.
I've done the same alpine trick for static binaries but may I introduce you to musl-cross-make? https://github.com/richfelker/musl-cross-make Just burned out static toolchains that make me static binaries for all architectures gcc supports. Much like musl.cc but they suggest building your own and I do. I use these toolchains on debian (/ anywhere a non-ancient linux kernel runs) to make static binaries, you can too!
When I looked into this, I decided against musl-cross-make because all the tools in there are way out of date. Due to its use of custom patches, you can't easily update it yourself; you must use their specific outdated versions. They're a full major version behind on gcc (it was two major versions behind at the time), and they're two minor versions behind on musl itself. Binutils is five minor versions behind. It was…
Not an apologist by any means and all that makes sense. I've found it easy and not-a-problem to bump those versions in the config file and made gcc 11.2.0 compilers with everything up-to-date this way quite recently.
There's also this fork of MCM that's a fair amount more active, as it backs musl.cc. I probably should have mentioned it
https://git.zv.io/toolchains/musl-cross-make
Regardless, your argument that a distro taking care of something for you is exactly what they're there for. Makes sense to me.
Mostly surfacing that there's multiple ways to handle that sort of thing. I find cross compiling simplest with these and version bumping wasn't a big deal at all compared to the complexities of other ways of cross compiling.
Re: Alpine Linux: Brilliant Linux Distro
#84I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues. eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...
Interesting! Thank you for sharing. Did you notice that some language are more affected than others? For example Go vs. Node.js?
Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.
Re: Alpine Linux: Brilliant Linux Distro
#85Earlier quoted context omitted.
Compiling is CPU-bound and doesn't really bottleneck on anything that libc is involved with other than the allocator, so unless musl's allocator is comically slower than glib's, chances are it's just -Os.
I believe musl’s allocator is known for being pretty poor.
musl-1.2.1.tar.gz (sig) - August 4, 2020
This release features the new "mallocng" malloc implementation, replacing musl's original dlmalloc-like allocator that suffered from fundamental design problems.
https://musl.libc.org/releases.html Rest of the detail in the actual changelog is worth reviewing too
Re: Alpine Linux: Brilliant Linux Distro
#86I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues. eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...
Also, isn't Python problematic with musl-libc [0]?
Re: Alpine Linux: Brilliant Linux Distro
#87Earlier quoted context omitted.
I've tried alpine's gcc before and it was about 50% slower to compile a bunch of packages. That said, if you compile llvm with MinSize/-Os on glibc distro's it runs 50% slower too, so it's unclear if slowdown is due to musl libc or rather that they optimize for size instead of performance.
Compiling is CPU-bound and doesn't really bottleneck on anything that libc is involved with other than the allocator, so unless musl's allocator is comically slower than glib's, chances are it's just -Os.
This is pretty much -Os.
Void Linux provides -O2 builds of GCC for both musl and glibc with a negligible performance difference. I have not benchmarked it lately, but it is clearly below 10%.
Re: Alpine Linux: Brilliant Linux Distro
#88Earlier quoted context omitted.
Yep. I had an (immutable) JVM ECS service that worked fine for a year, then started failing DNS in just one of several AWS AZs and for just one of various host names--one with something like 24 A records, a few but not an outrageous number. Occasionally forking a process to run 'dig' on the same name made it work for a little while. AWS support's only advice was "don't use Alpine"; annoyingly, switching the container…
Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.
Yes, my theory was also some DNS variance in the AZ, but AWS stolidly refused to supply any information to that effect. Adding 'dig +trace' to the equation only deepened the mystery.
Re: Alpine Linux: Brilliant Linux Distro
#89Earlier quoted context omitted.
When I looked into this, I decided against musl-cross-make because all the tools in there are way out of date. Due to its use of custom patches, you can't easily update it yourself; you must use their specific outdated versions. They're a full major version behind on gcc (it was two major versions behind at the time), and they're two minor versions behind on musl itself. Binutils is five minor versions behind. It was…
Not an apologist by any means and all that makes sense. I've found it easy and not-a-problem to bump those versions in the config file and made gcc 11.2.0 compilers with everything up-to-date this way quite recently. There's also this fork of MCM that's a fair amount more active, as it backs musl.cc. I probably should have mentioned it https://git.zv.io/toolchains/musl-cross-make Regardless, your argument that a dist…
I don't think I had seen that fork before, thanks for the link. They appear to be much more up-to-date. There are still issues using Alpine as a development container. I ultimately resorted to a separate Ubuntu container for development, building against glibc. It was the only way to get VSCode's C++ and debugger extensions to work on aarch64. I don't love the workaround; using Ubuntu for everything would be nicer. I only use Alpine for test and release builds.
Re: Alpine Linux: Brilliant Linux Distro
#90I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues. eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...
The person responsible for musl's DNS support thinks broken DNS is good UX and purposefully didn't add tcp upgrade support to it.
Note that some DNS resolvers do not provide truncated UDP results apparently so that might explain some of the weird DNS issues people see https://twitter.com/RichFelker/status/994629795551031296