It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens).
Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).
How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
1) Why should I limit myself to the available APIs?
2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.
I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ ABI all depend into each other. You cannot change any of them independently. All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library and make calls like dlopen.
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
In what sense do binary interpreter / loader, GCC compiler, C library and system C/C++ ABI dependent on each other? I have certainly mixed different versions of all these components without problems so far.
When you compile GCC you need to provide a full glibc installation as your target. It is also a dependency of libstdc++.
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
I can't comment on the C++, I can imagine there plenty of issues, but for C I don't see this. You need some libc if you compile with gcc, but this generally does not introduce a hard version dependency on the specific version (there may be a minimum requirement if you compile against a new version that a symbol with a different ABI).
I don’t imagine that you’re unaware of any of this, but: ld.so and libc.so are heavily interdependent in deliberately undocumented ways with Glibc and outright the same file with shared Musl. And while you might usually get away with using any old GCC with the right architecture and ABI (especially for C; cf the musl-gcc hack), technically it needs to be built to target a specific libc version (particularly via symbol versioning; I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases). The C compiler of course is tied to the C ABI, and this wouldn’t be really worth mentioning except for the time where the GCC devs accidentally the whole SysV i386 ABI and pretended that the stack was always 16-byte aligned, why do you ask, except on RHEL. The C++ parts I can’t really comment on.
I am not really sure. For ld.so and libc.so I may believe this. The C ABI is very stable, and if you use a new symbol from a newer glibc, you certainly depend on it, but this can also be avoided. In any case, I do not see what is fundamentally misdesigned here. I can't quite image how it could work differently. If you upgrade something so that the e ABI changed you natually need to update other components. Static linking certainly seems a very poor replacement for this.
The interpreter/loader is glibc and a key part of bootstrapping an executable built against glibc is loading libc itself before continuing on to load the program. Versioning is a problem when distributing binaries linked against a newer glibc to distros that ship an older one. The C compiler doesn't really care as much.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well.
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
There is no libc independent ABI. ABI doesn't purely mean just calling conventions.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
While I don't disagree with some of the pain you describe, you conveniently gloss over the fact that gnu developed a system that worked, and then made it free to everyone to consult and use.
BSD also did it. They did it better. Maybe more modern but AOSP also did it but at a different level of binary: instead of ELF, using compiled Java bytecode archives.
> all shared libraries depend on the specific glibc version to load them
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
It is not just compatibility. You cannot load them into the memory with your system dynamic loader. You need to also ship ld-linux.so with the new version of glibc you have, if you were to distribute your program independently.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
Well, now it's possible! Furthermore, SoLo binaries can run, without modification, on glibc-based distros, alpine, and soon on android/bionic (not committed yet).
On the one hand, this is technically true, but on the other, what serious issues do you know that will cause problems in practice? I run tests on 1,000 of the most popular Debian packages.
If you want to load the OpenGL/Vulkan vendor driver then unfortunately you don’t have much of a choice: those are linked against glibc, and I believe generally also against libwayland so screw off if you want a different protocol library (I might be wrong about the latter part). If you instead want to load plugins or whatnot into your statically linked executable, then personally I’d argue that you shouldn’t be emulating Linux dynamic linking semantics at all, because the whole late-bound global namespace thing is silly and wrong. (Solaris, which is where Glibc took this model from, moved away from it[1] as much as compatibility allowed, and so did Darwin[2], whereas Windows never made the mistake to begin with, but Glibc persisted and Musl copied it.)
Mapping parts of files into executable memory, and then executing them, had better be bulletproof! Exploiting this seems like a direct path to RCE, and it's likely that this sort of library is used by privileged code.
Purely academically, this is a very cool piece of code! Just hoping that it gets a thorough vetting before used by privileged/security-critical software :)
Well, ld.so already does this, and it's no big deal. The Python interpreter also does this when executing a .py script (code is code, whether it's machine-readable or human-readable).
In any case, we take testing very seriously—every glibc shim we've written is covered with tests, and we run our loader against 1000 of the most popular Debian packages. The project has 100% code coverage. Perhaps, if I have the time, I'll also do some fuzzing on this thing.
How are we supposed to take this stuff seriously if the author (sic) isn't even willing to write the readme? Claude exists! If I want some slop I can push the button myself.
README.md was written by me, and I, of course, used claude/codex for it. In general, I do everything through claude/codex, the reasons are described in https://github.com/pg83/solo/blob/main/CONTRIBUTING.md . And no, it's not low effort, and no, I don't see the point in wasting time de-claude-ifying the text just to avoid it looking like I didn't spend enough time on it.
I believe usually when someone complains about text written by a language model they are hoping to read human-written text instead of human-laundered LLM output.
You make it clear why you write all your code through a llm. But a README is not code. Presumably you would like people to read it. A machine authored readme reflects poorly on a project.
In any case, it's open source. If you don't like something, even if the project seems generally useful, go ahead and fix it. The PR came in. I'm an engineer and I can write good code, but that doesn't mean I can write good README.mds!
The author of the README is me, the machine just wrote it.
I am not a native speaker of English, my written English is simply terrible, no one wants to read the text that I wrote exactly :))
It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens).
Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).
How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?
I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
It's a pretty poor solution, to be honest.
1) Why should I limit myself to the available APIs?
2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.
I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
https://bugzilla.redhat.com/show_bug.cgi?id=638477 is the most "famous" example.
There are also much less well-known "little things" that regularly pop up here and there.
> If your CI uses an older glibc you should be fine AFAIK.
In any case, my binaries work not only under glibc, but also under Alpine, and (work in progress) under android/bionic.
according to appimage recommendations as long as you build against glibc with an earlier version than the system it's run on it should be fine.
https://docs.appimage.org/reference/best-practices.html
I hear you about WINE though.
How this differs (is better!) from prior art - https://github.com/pg83/solo#how-this-differs-from-prior-wor...
I don't know much about musl.
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ ABI all depend into each other. You cannot change any of them independently. All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library and make calls like dlopen.
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
> system C/C++ ABI
C++ abi should not be included in this. It is independent from the other pieces and historically a source of incompatibility on its own.
Saying "C/C++ abi" as if they are the same is looney tunes, the former is very simple and stable and the latter is very complex.
See: https://news.ycombinator.com/item?id=49355262
How libstdc++ initializes global variables absolutely depends on glibc and ld-linux.so. That is part of C++ ABI.
In what sense do binary interpreter / loader, GCC compiler, C library and system C/C++ ABI dependent on each other? I have certainly mixed different versions of all these components without problems so far.
When you compile GCC you need to provide a full glibc installation as your target. It is also a dependency of libstdc++.
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
I can't comment on the C++, I can imagine there plenty of issues, but for C I don't see this. You need some libc if you compile with gcc, but this generally does not introduce a hard version dependency on the specific version (there may be a minimum requirement if you compile against a new version that a symbol with a different ABI).
For example, the itanium unwind ABI implementation lies between these three entities.
I don’t imagine that you’re unaware of any of this, but: ld.so and libc.so are heavily interdependent in deliberately undocumented ways with Glibc and outright the same file with shared Musl. And while you might usually get away with using any old GCC with the right architecture and ABI (especially for C; cf the musl-gcc hack), technically it needs to be built to target a specific libc version (particularly via symbol versioning; I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases). The C compiler of course is tied to the C ABI, and this wouldn’t be really worth mentioning except for the time where the GCC devs accidentally the whole SysV i386 ABI and pretended that the stack was always 16-byte aligned, why do you ask, except on RHEL. The C++ parts I can’t really comment on.
I am not really sure. For ld.so and libc.so I may believe this. The C ABI is very stable, and if you use a new symbol from a newer glibc, you certainly depend on it, but this can also be avoided. In any case, I do not see what is fundamentally misdesigned here. I can't quite image how it could work differently. If you upgrade something so that the e ABI changed you natually need to update other components. Static linking certainly seems a very poor replacement for this.
The interpreter/loader is glibc and a key part of bootstrapping an executable built against glibc is loading libc itself before continuing on to load the program. Versioning is a problem when distributing binaries linked against a newer glibc to distros that ship an older one. The C compiler doesn't really care as much.
> The C compiler doesn't really care as much.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
It depends on functions defined in a standardized ABI.
You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well.
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
There is no libc independent ABI. ABI doesn't purely mean just calling conventions.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
While I don't disagree with some of the pain you describe, you conveniently gloss over the fact that gnu developed a system that worked, and then made it free to everyone to consult and use.
BSD also did it. They did it better. Maybe more modern but AOSP also did it but at a different level of binary: instead of ELF, using compiled Java bytecode archives.
> all shared libraries depend on the specific glibc version to load them
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
See https://news.ycombinator.com/item?id=49355262 .
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
That's what I'm saying though, glibc-linked binaries are forwards (but not backwards) compatible.
It is not just compatibility. You cannot load them into the memory with your system dynamic loader. You need to also ship ld-linux.so with the new version of glibc you have, if you were to distribute your program independently.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
musl has no problem building and using shared libraries.
What you can't do is build something statically with musl and then reliably dlopen shared libraries built with glibc.
Well, now it's possible! Furthermore, SoLo binaries can run, without modification, on glibc-based distros, alpine, and soon on android/bionic (not committed yet).
musl does not perfectly emulate all aspects of glibc, so trying to use libraries that assume glibc can sometimes lead to problems.
On the one hand, this is technically true, but on the other, what serious issues do you know that will cause problems in practice? I run tests on 1,000 of the most popular Debian packages.
Mostly about precompiled libraries (proprietary software) and libraries and software that use GNU extensions.
Why not just pass the GPU to a docker container?
Available in README - https://github.com/pg83/solo#how-this-differents-from-prior-...
So not completely static, since it must link against a libc :P
The binary itself is completely static; the link even provides commands on how to check this!
> backed by its own ELF loader (x86-64 and aarch64) and a glibc ABI bridge
Yacks
If you want to load the OpenGL/Vulkan vendor driver then unfortunately you don’t have much of a choice: those are linked against glibc, and I believe generally also against libwayland so screw off if you want a different protocol library (I might be wrong about the latter part). If you instead want to load plugins or whatnot into your statically linked executable, then personally I’d argue that you shouldn’t be emulating Linux dynamic linking semantics at all, because the whole late-bound global namespace thing is silly and wrong. (Solaris, which is where Glibc took this model from, moved away from it[1] as much as compatibility allowed, and so did Darwin[2], whereas Windows never made the mistake to begin with, but Glibc persisted and Musl copied it.)
[1] https://www.linker-aliens.org/blogs/rie/entry/direct_binding...
[2] https://web.archive.org/web/20011004090044/http://developer....
Please elaborate and explain to people with less knowledge why this is bad.
Mapping parts of files into executable memory, and then executing them, had better be bulletproof! Exploiting this seems like a direct path to RCE, and it's likely that this sort of library is used by privileged code.
Purely academically, this is a very cool piece of code! Just hoping that it gets a thorough vetting before used by privileged/security-critical software :)
Well, ld.so already does this, and it's no big deal. The Python interpreter also does this when executing a .py script (code is code, whether it's machine-readable or human-readable).
In any case, we take testing very seriously—every glibc shim we've written is covered with tests, and we run our loader against 1000 of the most popular Debian packages. The project has 100% code coverage. Perhaps, if I have the time, I'll also do some fuzzing on this thing.
similar energy to my https://github.com/lunixbochs/crossldso
Oh, cool, another prior art I didn't know :)
How are we supposed to take this stuff seriously if the author (sic) isn't even willing to write the readme? Claude exists! If I want some slop I can push the button myself.
Tell me, are there any substantive comments on the text, on what has been done, and on the technical implementation, and not on the form?
Why would I ask you about this work? The root of my question is why is this type of output shared on github, rather than the inputs?
Rough
Isn't this a README? https://github.com/pg83/solo/blob/main/README.md
Parents point is the readme was written by Claude. Signals low effort.
README.md was written by me, and I, of course, used claude/codex for it. In general, I do everything through claude/codex, the reasons are described in https://github.com/pg83/solo/blob/main/CONTRIBUTING.md . And no, it's not low effort, and no, I don't see the point in wasting time de-claude-ifying the text just to avoid it looking like I didn't spend enough time on it.
I believe usually when someone complains about text written by a language model they are hoping to read human-written text instead of human-laundered LLM output.
You make it clear why you write all your code through a llm. But a README is not code. Presumably you would like people to read it. A machine authored readme reflects poorly on a project.
In any case, it's open source. If you don't like something, even if the project seems generally useful, go ahead and fix it. The PR came in. I'm an engineer and I can write good code, but that doesn't mean I can write good README.mds!
The author of the README is me, the machine just wrote it. I am not a native speaker of English, my written English is simply terrible, no one wants to read the text that I wrote exactly :))
Implicit assumption of gp is that the README is llm authored. (Which, I agree is how it reads to me.)