On The Other Hand · Undefined symbol: "stat@FBSD

Undefined symbol: "stat@FBSD_1.5"

Date: 2019-02-02 · Word Count: 386 · Reading Time: 2 minutes

While upgrading from FreeBSD 11 to 12 this weekend I came across a rather surprising error in the logs.

warning: connect to Milter service unix:/var/run/milter-regex/sock: No such file or directory

Odd, worked before the upgrade, let’s check it out:

$ sudo service milterregex restart
milterregex not running? (check /var/run/milter-regex/milter-regex.pid).
Starting milterregex.
ld-elf.so.1: /usr/lib/libmilter.so.6: Undefined symbol "stat@FBSD_1.5"
/usr/local/etc/rc.d/milterregex: WARNING: failed to start milterregex

What’s it using?

$ ldd /usr/local/libexec/milter-regex
/usr/local/libexec/milter-regex:
        libmilter.so.6 => /usr/lib/libmilter.so.6 (0x800250000)
        libthr.so.3 => /usr/lib/libthr.so.3 (0x800263000)
        libc.so.7 => /usr/lib/libc.so.7 (0x80048b000)

$ ldd /usr/lib/libmilter.so
/usr/lib/libmilter.so:
        libc.so.7 => /lib/libc.so.7 (0x800248000)

$ ldd /usr/lib/libmilter.so.6
/usr/lib/libmilter.so.6:
        libc.so.7 => /lib/libc.so.7 (0x800248000)

$ ldd /usr/local/libexec/milter-regex 
/usr/local/libexec/milter-regex:
        libmilter.so.6 => /usr/lib/libmilter.so.6 (0x800250000)
        libthr.so.3 => /lib/libthr.so.3 (0x800263000)
        libc.so.7 => /usr/lib/libc.so.7 (0x80028e000)

Hmm… Looks legit. Let’s check out the differences between the base OS and the jail (only one jail is having issues):

$ sha512 /lib/libc.so.* /jail/*/lib/libc.so.* /usr/lib/libc.so.* /jail/*/usr/lib/libc.so.*
SHA512 (/lib/libc.so.7) = 36cacff96ee9afc0c385e97fced07233149029eae95cfece7e527f4710e4c76017ee56d58a733b198777ea0516973d1bbc75937352354c96464037cd4fe2a472
...
SHA512 (/jail/${trouble}/lib/libc.so.7) = 36cacff96ee9afc0c385e97fced07233149029eae95cfece7e527f4710e4c76017ee56d58a733b198777ea0516973d1bbc75937352354c96464037cd4fe2a472
SHA512 (/jail/${trouble}/usr/lib/libc.so.7) = f387857e720e058c7ca60903cc55e3b195cdfbf4a665f879e837fc829a93b77fc03bef7b09c8095111c02ce3fc54e7c68b6f79307b5dc27bf3ac549bb934140c

Looks like the old libraries may not have been cleaned up properly. Let’s clean them up.

First, snapshot /usr/src and mount it into the jail:

$ sudo zfs snapshot zroot/usr/src@12.0p2

$ sudo zfs clone -o mountpoint=/jail/${trouble}/usr/src \
  zroot/usr/src@12.0p2 zroot/jail/${trouble}/usr-src

Then, login to the jail and see what it finds to clean up:

$ cd /usr/src

$ sudo make delete-old-libs
>>> Removing old libraries
Please be sure no application still uses those libraries, else you
can not start such an application. Consult UPDATING for more
information regarding how to cope with the removal/revision bump
of a specific library.
remove /usr/lib/libbsnmptools.so.0? y
remove /usr/lib/libbsnmptools.so? y
remove /usr/lib/libelf.so.2? y
remove /usr/lib/libxo.so.0? y
remove /usr/lib/libnv.so.0? y
remove /usr/lib/libcasper.so.0? y
>>> Old libraries removed

$ sudo make delete-old
>>> Removing old files (only deletes safe to delete libs)
remove /usr/share/man/man2/cap_rights_get.2.gz? y
remove /usr/include/pcap-int.h? y
remove /usr/include/c++/v1/__undef___deallocate? y
remove /usr/include/c++/v1/tr1/__undef___deallocate? y
remove /usr/include/netinet/ip_ipsec.h? y
remove /usr/include/netinet6/ip6_ipsec.h? y
remove /usr/lib/libbsnmptools.a? y
remove /usr/lib/libbsnmptools_p.a? y
remove /usr/lib/clang/3.8.0/include/sanitizer/allocator_interface.h? y
remove /usr/lib/clang/3.8.0/include/sanitizer/asan_interface.h? y
remove /usr/lib/clang/3.8.0/include/sanitizer/common_interface_defs.h? y
...

Looks like quite a bit.

Cleanup the extra mounts:

$ sudo zfs destroy zroot/jail/${trouble}/usr-src

$ sudo zfs destroy zroot/usr/src@12.0p2

And see how it worked

$ sudo service milterregex start
Starting milterregex.

Phew. Done.

So, if you happen to come across the rather esoteric Undefined symbol error after an upgrade, chances are that some old libraries are lying around and your application is linking against them incorrectly.

OTOH