An upgrade on Debian Testing last week broke my multiarch libc6 install (library cache fully broken, as described here).
I’m using a debian testing live cd from here, using the same architecture (amd64) as the system i want to recover.
Since my fileystem is an encrypted lvm, it has to be decrypted first.
Check the partition (first is /boot)
# fdisk -l /dev/sda # cryptsetup luksOpen /dev/sda2 sda2_crypt # apt-get install lvm2 # lvdisplay
Prepare the chroot (mount sda1 boot as well. root and home are required, the other mounts are just for backup reasons)
# cd / # mount /dev/sda1 /mnt/boot # mount /dev/mapper/luksvg-root /mnt # mount /dev/mapper/luksvg-home /mnt/home # mount /dev/mapper/luksvg-data /mnt/data # mount /dev/mapper/luksvg-vms /mnt/vms # mount /dev/mapper/luksvg-backup /mnt/backup
Do a backup, if not done yet.
Mount all required for system chroot.
# mount -t proc /proc /mnt/proc # mount -t sysfs /sys /mnt/sys # mount -o bind /dev /mnt/dev # mount -t devpts /dev/pts /mnt/dev/pts
In order to resolve domain names, copy over the resolv.conf
# cp -L /etc/resolv.conf /mnt/etc/resolv.conf
Normally I would have gone for /bin/bash, but since this already fails due to the broken libs, I am choosing /bin/sh
# chroot /mnt /bin/bash /bin/bash: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory # chroot /mnt /bin/sh #
Now for the fun part – our system is really really broken, commands like ‘ls’ or tab completion does not work!
Try to rebuild the library config cache.
# /sbin/ldconfig
We should have recovered the libraries. In order to fix the entire system from the previously broken state, it’s more difficult.
[insert 2 hours here]
Well, tbh, it wasn’t possible. The entire thing was creating a dependency loop, but nailed down to libc6:amd64 and libc6:i386 which always wipe the ld.so.cache file in their preinst script.
Fixing that would require deep libc6 knowledge as well as proper debian package rebuilding on libc6.
I’ll quit on this, and re-install the system.
if dpkg --compare-versions "$preversion" lt 2.13-38; then # upgrading from a pre-multiarch libc to a multiarch libc; we have # to blow away /etc/ld.so.cache, otherwise the old unpacked libc # is still first in the cache and segfaults when combined with # our newly-unpacked ld.so. Do this last to avoid slowing down the # rest of the upgrade. Version number bumped to 2.13-38 to also # cover cache format upgrades for ARM. rm -f /etc/ld.so.cache fi
Below is a copied log from some attempts. Whoever may get lucky.
root@debian:/# chroot /mnt /bin/sh $ /sbin/ldconfig $ ls backup boot dev gmon.out initrd.img lib32 lost+found media opt root sbin srv tmp var vms bin data etc home lib lib64 man mnt proc run selinux sys usr vmlinuz $ bash debian / # ls backup/ boot/ dev/ gmon.out initrd.img@ lib32/ lost+found/ media/ opt/ root/ sbin/ srv/ tmp/ var/ vms/ bin/ data/ etc/ home/ lib/ lib64/ man/ mnt/ proc/ run/ selinux/ sys/ usr/ vmlinuz@ debian / # apt-get -f install Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following packages were automatically installed and are no longer required: google-gadgets-common google-gadgets-gst google-gadgets-qt libdns81 libevtlog0 libfs6 libggadget-1.0-0b libggadget-qt-1.0-0b libisc83 libmongo-client0 libnet1 libsyslog-ng-3.3.5 menu-xdg plasma-scriptengine-googlegadgets x11-apps x11-session-utils x11-xfs-utils xinit xorg Use 'apt-get autoremove' to remove them. The following extra packages will be installed: libc-bin libc-dev-bin libc6 libc6:i386 libc6-dbg libc6-i686:i386 Suggested packages: glibc-doc glibc-doc:i386 locales:i386 The following packages will be upgraded: libc-bin libc-dev-bin libc6 libc6:i386 libc6-dbg libc6-i686:i386 6 upgraded, 0 newly installed, 0 to remove and 24 not upgraded. 8 not fully installed or removed. Need to get 0 B/13.6 MB of archives. After this operation, 6,144 B disk space will be freed. Do you want to continue [Y/n]? Reading changelogs... Done Preconfiguring packages ... (Reading database ... 185753 files and directories currently installed.) Preparing to replace libc6-dbg:amd64 2.13-37 (using .../libc6-dbg_2.13-38_amd64.deb) ... Unpacking replacement libc6-dbg:amd64 ... Preparing to replace libc-bin 2.13-37 (using .../libc-bin_2.13-38_amd64.deb) ... Unpacking replacement libc-bin ... Processing triggers for man-db ... Setting up libc-bin (2.13-38) ... (Reading database ... 185753 files and directories currently installed.) Preparing to replace libc6:amd64 2.13-37 (using .../libc6_2.13-38_amd64.deb) ... De-configuring libc6:i386 ... Unpacking replacement libc6:amd64 ... dpkg-deb: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory dpkg: error processing /var/cache/apt/archives/libc6_2.13-38_amd64.deb (--unpack): subprocess dpkg-deb --fsys-tarfile returned error exit status 127 dpkg-deb: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory dpkg: error processing /var/cache/apt/archives/libc6_2.13-38_i386.deb (--unpack): subprocess dpkg-deb --control returned error exit status 127 Errors were encountered while processing: /var/cache/apt/archives/libc6_2.13-38_amd64.deb /var/cache/apt/archives/libc6_2.13-38_i386.deb E: Sub-process /usr/bin/dpkg returned an error code (1) sed: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory debian / # ls ls: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory sed: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory debian / # /sbin/ldconfig # cd /var/cache/apt/archives # dpkg -i --force-depends libc6-amd64_2.13-38_i386.deb # dpkg -i --force-depends libc6-i386_2.13-38_amd64.deb # dpkg -i --force-depends libc6-i686_2.13-38_i386.deb # dpkg -i --force-depends libc-bin_2.13-38_amd64.deb # dpkg -i --force-depends libc6-dev-amd64_2.13-38_i386.deb # dpkg -i --force-depends libc6-dev_2.13-38_i386.deb # dpkg -i --force-depends libc-dev-bin_2.13-38_amd64.deb debian /var/cache/apt/archives # dpkg -i --force-depends libc6_2.13-38_i386.deb (Reading database ... 185753 files and directories currently installed.) Preparing to replace libc6:i386 2.13-37 (using libc6_2.13-38_i386.deb) ... De-configuring libc6:amd64 ... Unpacking replacement libc6:i386 ... dpkg-deb: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory dpkg: error processing libc6_2.13-38_i386.deb (--install): subprocess dpkg-deb --fsys-tarfile returned error exit status 127 Errors were encountered while processing: libc6_2.13-38_i386.deb sed: error while loading shared libraries: libselinux.so.1: cannot open shared object file: No such file or directory
I had exactly the same problem and managed to fix this, doing the following:
– boot from rescue cd into installer environment
– mount root partition (/target)
– execute: /target/sbin/ldconfig -r /target
– chroot to /target
– provide apt with library path to that it can continue:
$ LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/ apt-get -f install
install process completed, ld.so.cache was recreated and everything is up running again.
Hope this helps!
oh. library path – what a heck of idea, thank you!
Can you post the commands?
Thanks