Comments
You can use your Mastodon account to reply to this post.
Pushing to far, too hard, in the wrong direction!
Today, I’ll digress on a Nix derivation for a proprietary package, namely Zed (free version) from Prim’X, that took a lot of tricks to get working. Further, I’ll detail the multiple paths I took and that turned out to be false leads.
TL;DR
You can find the derivation (package) on my Github repository here with instruction on how to use the repository here.
Motivation
Zed is a tool for encrypting/decrypting sensitive data using a proprietary algorithm from Prim’X. When I left my previous work, I purposefully kept a Zed archive with unimportant files and I held on to this archive so I could check if I could open it on Linux systems after a few years.
Why would I do that? Well, Zed has been certified and qualified by ANSSI. So it means that a few French administrations were bound to use it and I’m curious to see where that product is going.
It turns out that I can still run the Linux binary, but free version of Zed is packaged only for Ubuntu (14.04 at the time). It is not statically compiled and thus depends heavily on the system. In my case, none of it worked out of the box with NixOS (the Linux distribution that I’m using).
I saw it not as a problem, but as a challenge and a good way to learn.
The story so far
During my initial packaging of Zed Free a few years ago, I went straight for the goal: I downloaded the Ubuntu package from the editor’s website (you can find it here) and unpacked it in the Nix derivation with dpkg-deb.
This straightforward approach was bound to fail for three reasons:
- missing libraries
- hardcoded path in the binaries: Nix stores everything in the path /nix/store and does not comply with the regular /usr/lib path.
- preparation script not being run
Each of these issues were fixable and thus were fixed. Here is how.
Missing libraries
This one is pretty simple: run the binary. It complains about a library: find the Nix derivation that contains it and add it to Zed derivation. Add all these library to a wrapper that LD_PRELOAD it all.
Hardcoded path
This is the trickiest one. As I said before, the Zed binary loads data (icons and stuff) from /usr/share/ (and also /etc/primx). These paths are hardcoded in the syscall of the binary. Because the Nix path are going to be longer (in the from of /nix/store/h5ffa3d8i5yfb18r56m95kf7mdf2mcmf-zed-free-2020.4/), you can’t do a simple sed with the binary.
After I few experiments, I set out to dynamically patch a few key syscalls using a preloaded (LD_PRELOAD) stub. The only purpose of that program is to patch the syscall (open, stat) and replace the hardcoded path with the one where the derivation will be installed. It’s only fair to say that I would not have made it without this [excellent post from Rafał Cieślak](https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-tricks-using-ld_preload-to-cheat-inject-features-and-invest igate-programs/).
The stub code can be found here.
Preparation script
This is a preparation script part of the postinst of the Ubuntu package:
#!/bin/sh
# this is mostly a rip-off from the debian package postinst
ACCESS_FILES_PATH=/tmp/primx/accessfiles
ALL_USERS_REGISTRY=/tmp/primx/users.registry
mkdir -p ${ACCESS_FILES_PATH} ; chmod 1777 ${ACCESS_FILES_PATH}
touch $ALL_USERS_REGISTRY
chmod 666 $ALL_USERS_REGISTRY
As you can see, it is not very pretty, but it is required. In the derivation, I make sure to include this script in a wrapper so that it can be run every time Zed is launched.
What changed?
It actually worked for quite a while but there wouldn’t be a blog post if nothing had changed. A system upgrade happened and two things changed:
- OpenLDAP got upgraded to version 2.6 (and it is a good thing)
- Zed got upgraded to a 2022.2 version (previous version 2020.4). It is now expecting to be installed on Ubuntu 18.04 instead of Ubuntu 14.04, so I expected a few dependencies to have changed.
First attempt to start after the upgrade
So first update tells me that the SHA sums has changed: I update with the new SHA sum so that the derivation can still be installed, intending to deal with the aftermath.
/home/tcheneau/.nix-profile/bin/zed_free: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory
What is it telling me? Zed wants OpenLDAP 2.4.
Going the hard and wrong way
The following section details multiple approaches I tried while trying to get the binary with the new OpenLDAP 2.6 library instead of the hardcoded 2.4. As you will see, it’s a brutal fail, with a few lessons learned along the way.
Approach one: symlink the new libraries with the name of the required ones
ln -s ${openldap}/lib/libldap.so $out/lib/libldap_r-2.4.so.2
ln -s ${openldap}/lib/liblber.so $out/lib/liblber-2.4.so.2
Here I make two symlinks with the name the binary expects. The ${openldap} variable is the OpenLDAP derivation, so you don’t have to know the path to your files in your Nix store.
$ zed_free
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/b29d16vvwf86kadgk075aqprs2fp5wzb-zed-free-2020.4/lib/liblber-2.4.so.2: version `OPENLDAP_2.4_2' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/b29d16vvwf86kadgk075aqprs2fp5wzb-zed-free-2020.4/lib/libldap_r-2.4.so.2: version `OPENLDAP_2.4_2' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
zed_free
It fails but it tells me something: it would have loaded, provided that the new library had to good version for exported function symbols.
Approach two: sedding the hardcoded symbol name
It was doomed to fail but quick to test, so I added the following line in the derivation:
sed -i "s/OPENLDAP_2\.4_2/OPENLDAP_2\.200/g" $out/bin/zed_free
New output that could be puzzling if I didn’t know what to expect:
$ zed_free
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/07pfzrg9qxqrmnlxyd1316an3cn57wyz-zed-free-2020.4/lib/liblber-2.4.so.2: version `OPENLDAP_2.200' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/07pfzrg9qxqrmnlxyd1316an3cn57wyz-zed-free-2020.4/lib/libldap_r-2.4.so.2: version `OPENLDAP_2.200' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
Here, the dynamic interpreter that loads library does not rely on the name to load a symbol (it actually relies on a hash). This approach was doomed from the start. Note that the error is now more misleading because it shows the “pretty” name of the symbol and not the one it is looking for.
Approach three: removing requirements on the binary with patchelf
Scratch everything before, I was in the wrong, I need to use a more powerful tool.
Happily, Nix’s community maintains a wonderful tool named patchelf. It can perform various tasks on an ELF binary: change the dynamic interpreter, change the name/path of a dynamic library, remove a symbol version. The last bit is intriguing.
Here, I tried a few parameters and the most conclusive one would have been:
${patchelf}/bin/patchelf \
--interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
--replace-needed liblber-2.4.so.2 liblber.so.2 \
--replace-needed libldap_r-2.4.so.2 libldap.so.2 \
$out/bin/zed_fre
That is, it rewrites the ELF data so that it loads the libraries with the OpenLDAP 2.6 paths. However it yields the following error now:
$ zed-free
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /home/tcheneau/.nix-profile/bin/zed_free: no version information available (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: symbol lookup error: /home/tcheneau/.nix-profile/bin/zed_free: undefined symbol: , version
This is because the binary was stripped. Thus adding the dontStrip = true; is required as an argument to stdenv.mkDerivation.
New output now:
$ zed-free
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/ay0a0phrkgmmwd70dxqbg9dfgg7bflp7-openldap-2.6.2/lib/liblber.so.2: version `OPENLDAP_2.4_2' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
/home/tcheneau/.nix-profile/bin/zed_free: /nix/store/ay0a0phrkgmmwd70dxqbg9dfgg7bflp7-openldap-2.6.2/lib/libldap.so.2: version `OPENLDAP_2.4_2' not found (required by /home/tcheneau/.nix-profile/bin/zed_free)
So it loads the right file (liblber.so.2) but still looks for the old symbols. It seems to relate to the following bug. I tried a few other options, but I could not successfully update the symbols to the new one with patchelf.
Patchelf was my best hope but it doesn’t seem to have the required functionality. Consequently, I wrote some throw away code to experiment a little more and edit the ELF data.
Approach four: modifying ELF header with Golang
As I trying to learn Golang, I set out with this language. Golang’s battery included provides an ELF parsing library in the form of the debug/elf library. Sadly, if it can parse the headers, it is of no use to edit the header.
After a few search, I set out to parse the headers myself, using the following ressources:
- struc: a great library that enables you to annotate structures and quickly Packing and Unpacking of binary data
- Wikipedia’s page on ELF
- Linux’s foundation page on Symbol Versioning
Header by header, I decoded the zed-free binary until I had this output (source code of this throw away code can be found here):
$ go run main.go ~/tmp/zed_free-wrapped
/tmp/go-build3477890213/b001/exe/main
.gnu.version_r
LittleEndian
&{c00010e7e0 0 0 7fffffffffffffff}
Reading bytes: 5272928
Elf Header: &{Magic:1179403647 Class:2 Data:1 Version:1 ABI:3 ABIVersion:0 Padding:[0 0 0 0 0 0 0] Type:2 Machine:62 Version2:1 Entry:4661664 PhOffset:64 SectionOffset:5270944 Flags:0 EhSize:64 ProgHeaderSize:56 ProgHeaderNum:11 SectionHeaderSized:64 SectionHeaderNum:31 SectionHeaderIdx:30}0: Magic int = 464c457f
1: Class uint8 = 2
2: Data uint8 = 1
3: Version uint8 = 1
4: ABI uint8 = 3
5: ABIVersion uint8 = 0
6: Padding []uint8 = 00000000000000
7: Type uint = 2
8: Machine uint = 3e
9: Version2 uint = 1
10: Entry uint = 4721a0
11: PhOffset uint = 40
12: SectionOffset uint = 506da0
13: Flags uint = 0
14: EhSize uint = 40
15: ProgHeaderSize uint = 38
16: ProgHeaderNum uint = b
17: SectionHeaderSized uint = 40
18: SectionHeaderNum uint = 1f
19: SectionHeaderIdx uint = 1e
0: Name uint = 5b
1: Type uint = 6ffffffe
2: Flags uint = 2
3: Addr uint = 41ece0
4: Offset uint = 1fce0
5: Size uint = 2f0
6: Link uint = 1
7: Info uint = 10
8: AddrAlign uint = 8
9: EntSize uint = 0
5271460
506fa4
0: Version uint = 1
1: Cnt uint = 1
2: File uint = 111d9
3: Aux uint = 10
4: Next uint = 20
conv_close
0: Hash uint = 2881502
1: Flags uint = 0
2: Other uint = 1f
3: Name uint = 110b3
4: Next uint = 0
At some point, I even had to recompute the symbol’s name hashes, so I quickly hack some C code to compute the hash of the OPENLDAP 2.6 library:
#include <stdio.h>
// from https://docs.oracle.com/cd/E19683-01/816-7777/6mdorm6jk/index.html#chapter6-48031
unsigned long
elf_Hash(const unsigned char *name)
{
unsigned long h = 0, g;
while (*name)
{
h = (h << 4) + *name++;
if (g = h & 0xf0000000)
h ^= g >> 24;
h &= ~g;
}
return h;
}
int main(int argc, char * argv[]) {
printf("%x\n", elf_Hash("OPENLDAP_2.4_2"));
printf("%x\n", elf_Hash("OPENLDAP_2.200"));
return 0;
}
I reached a point where I had enough information to locate key data in the ELF headers and replace it in the binary programmatically. I did not recorded the output, but suffice to say that the ELF headers contain not only the name of the symbol, but the version and the hash of the name, in multiple places. Having disabled a few headers, replaced a few data and observing that zed-free was still looking up for the old symbols, I gave up and decided to give it a rest. The way forward would have been to implemented more headers and data structure.
Coming back to my sense
After a little break, I started to rethink my approach: I was trying to trick Zed into accepting OpenLDAP 2.6 library. This approach is not harnessing the power of Nix. That is: I could provide OpenLDAP 2.4 only for the Zed derivation and still use 2.6 for every other derivations.
So I set out to resurrect OpenLDAP 2.4, that was pretty straightforward:
- Clone nixpkgs repository
- Read the git log of the pkgs/development/libraries/openldap/default.nix file
- Extract the version before the OpenLDAP 2.6 upgrade
- Add this file to my custom nixpkgs repository (under the name openldap24).
And voilà:
Alternatively, I could have imported an earlier nixpkgs tree and get the openldap package from it. This probably would have been cleaner but I spent enough time on the topic already.
Capitalizing on a loss
A few tricked learned along the way:
- zeroing a few bytes from a binary (here it removes the .gnu.version_r header):
# zero the 4 bytes of the ELF .gnu.version_r Section Header (becomes unused entry)
dd if=/dev/zero of=$out/bin/zed_free bs=1 count=4 seek=5271460 conv=notrunc
- binary patching with xxd and sed (here this is a patch for changing an ELF symbol hash in a program):
cat $out/bin/zed_free|${vim}/bin/xxd|sed 's/0001ecf0: 0215 8802/0001ecf0: 1018 8802/'|${vim}/bin/xxd -r > zed_free
chmod +x zed_free
mv zed_free $out/bin/zed_free
Closing words
Nix is an interesting package manager and NixOS is my current distribution. I’m not planning to change that anytime soon. We can see that vendors favor mainstream distributions (and it’s already good that there is a Linux binary at all) and that Ubuntu packages won’t work well out of the box on NixOS: this is to be expected.
Overall, I’m quite satisfied that I can have two version of OpenLDAP on my system, build a declarative package, ship it with all the tweaks I want and integrate it to my distribution that easily.
P.S: I wanted to play with ELF headers for quite some time, so it all was a good learning opportunity.