Surviving Ubuntu's perpetual out of space boot partition

I'm somewhat guilty of just using the default partitioning schemes when installing recent versions of Ubuntu (currently 16.04 LTS). I don't want to bother with manually setting up LVMs and encrypted partitions and segregating log data from user data and all of that. I just need a root partition and I'll handle everything else on my own. Unfortunately, that means Ubuntu gives you a <256MB /boot partition. That's not enough.

Something like this happens while going about your business upgrading your system:

update-initramfs: Generating /boot/initrd.img-4.4.0-57-generic

gzip: stdout: No space left on device
E: mkinitramfs failure find 141 cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-4.4.0-57-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-extra-4.4.0-57-generic (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-image-extra-4.4.0-57-generic; however:
  Package linux-image-extra-4.4.0-57-generic is not configured yet.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured

What's going on?

If you're keeping up with your regular apt updates, you're usually keeping around four or so kernels on your boot partition. Generally, that includes the initial kernel that was installed with the system and three of the most recent kernels (and if you haven't restarted in awhile, your currently running kernel too). The problem with that is that you're now basically out of space.

Being out of space doesn't really manifest itself as a problem until you go to update your system again. apt will dutifully download the packages, install the kernel, library modules, header files, and whatever else you've installed/required. But usually by the time it goes to rebuild the initramfs images, you'll find that you're out of space on your boot partition and that causes things to go south. Hopefully, it's only the initramfs steps that have been caught by this, but sometimes additional packages that were in the queue to be installed or updated get caught in the crossfire.

Usually at this point you can run apt autoremove to reclaim some space from a now-obsolete kernel and then manually reinstall the failed new kernel packages. Sadly, being on top of running auto-remove before updating doesn't necessarily guarantee you'll avoid this.

What can you do?

1. Periodically running apt autoremove might not help.
2. There does not appear to be a way to configure a number of installed kernels.
3. Repartitioning a more-or-less production system is a headache.

Keeping /boot cruft free

What's left is using a few ways of keeping the boot partition slimmed down. The first is using a script called purge-old-kernels to minimize the number of installed kernels, and the other is adjusting how initramfs-tools creates the initrd images.

purge-old-kernels

This script created by Dustin Kirkland happened to be installed by default on my machine as part of the byobu package.¹ The author goes into more detail on his blog about what the utility is and does, and what to do if you're not running 16.04+. It is as simple as running:

sudo purge-old-kernels

The two things we need to know are that it won't attempt to remove a running kernel and it (by default) keeps two additional kernels. Periodically running this (or maybe even just running it before a kernel package upgrade) should keep enough space available.

Trimming initrd images

You'll probably find that the biggest disk space offender in your boot partition are the initrd images that Linux uses when it's booting up to speed up the loading of modules and other things it might need to boot your system (such as more esoteric file system or network drivers).²

Changing initramfs options to only use the modules our kernel is using, and using a (slightly) more efficient compression algorithm will also significantly slim down those images. On Ubuntu, we can edit /etc/initramfs-tools/initramfs.conf and then rebuild the images.

The two lines to edit in initramfs.conf are:

MODULES=dep    # Rather than MODULES=most
COMPRESS=lzma  # Rather than COMPRESS=gzip

You can choose any of the listed compression algorithms, I just happened to pick lzma. Once you've edited the configuration file, you'll need to rebuild the images. Though, if you're paranoid like me, you'll run uname -a to figure out what your currently running kernel version is, and copy its initrd image to a safe place, just in case.

sudo update-initramfs -u -k all

On my machine, doing those two things dropped individual image sizes from 35MB to around 9MB. Multiply those space savings by how many kernels you happen to have installed, and that can be nearly half of your default partition size. A quick reboot to ensure that your new initrd images are working just fine is all that's need to finish this task up.