Preparing a Debian 10 netinst USB stick with preseeding and UEFI support

Introduction

Putting the stock Debian 10 netinst image onto a USB stick is straightforward and well documented. However, if you want to provide answers to the questions that the Debian installer asks before it asks them – a process called preseeding – then it gets more complicated.

Firstly, at the time of writing, the official documentation regarding preparing a bootable USB stick does not mention UEFI or preseeding.

Secondly, preseeding is a slow and frustrating process:

the official documentation for preseeding is only slightly more detailed than the official example configuration file,
each test installation takes several minutes so development is slow,
there is not quite a one-to-one mapping between the questions that can be asked and what can be preseeded.

Thirdly, the environment in which the Debian installer runs is very restricted: most normal commands (e.g. bash) are missing; some normal commands are implemented directly in the BusyBox shell but with limited functionality (e.g. grep). There are some ‘helper’ scripts for the Debian installer (e.g. list-devices, debconf-set) but these appear to be undocumented.

This article is my attempt to document how to prepare a Debian 10 netinst image, suitable for a USB stick or CD, suitable for a real machine or a VM, preseeding answers for the Debian installer and with UEFI support. Note that the image that this procedure produces does not have legacy boot support (because I did not need legacy boot support).

My approach to addressing the difficulties described above is as follows:

during the installation, I get the Debian installer to do the absolute minimum;
after the installation finishes, I run a site-specific script to do all the rest.

For example, during the installation, I tell the Debian installer not to create normal users, but just to set root’s password. After the installation finishes, I run my site-specific script, which installs and configures LDAP and mounts /home from NFS fileservers. Also, during the installation, I tell the Debian installer not to create swap space. After the installation finishes, I run my site-specific script, which creates an logical volume, formats it as swap space and adds an entry for it to /etc/fstab.

In this article, the usual rules about fonts apply: text in a fixed width font represents I/O or OS commands; text in a bold fixed width font represents user input.

What we will need

We will need the following items:

USB stick of size 250MB or bigger (if you intend to use the USB stick image only to install VMs, then you will not need a USB stick).
a Linux machine with root access and an internet connection on which to perform the procedure.

Creating an empty USB stick image

A USB stick image is a file that will be copied to a USB stick. But it is not to be copied into a filesystem that resides in a partition on the USB stick; instead the USB stick image is to overwrite the entire USB stick. This is because the USB stick image contains a boot record, a partition table and a partition containing a filesystem, all of which are meant to replace those already on the USB stick.

This means that when preparing the USB stick image we will need to partition it and mount the partitions in order to populate them. This presents a problem: although parted can partition a USB stick image, how are we then to expose the partitions in order to mount them? We need a way to make a file behave more like a disk-like device: this can be done using a loop device.

We record the location and the size of the USB stick image that we will create in shell variables:

USB_IMAGE_FILE=/tmp/usb.img           #  USB stick image file
USB_IMAGE_SIZE_MB=250                 #  USB stick image size in MB

We use dd to create the empty USB stick image:

dd if=/dev/zero of=$USB_IMAGE_FILE bs=1M count=$USB_IMAGE_SIZE_MB

We determine the first loop device that is not in use with:

comm -1 -3 <(losetup -a | sed 's/:.*//') <(ls /dev/loop[0-9]) | head -1

We record which loop device to use in a shell variable:

USB_IMAGE_LOOPDEV=<loop-device>       # e.g. USB_IMAGE_LOOPDEV=/dev/loop0

We make the file behave like a device:

losetup $USB_IMAGE_LOOPDEV $USB_IMAGE_FILE

Partitioning the USB stick image

According to Arch Linux’s wiki, it is recommended that the partition table type is GPT, since old firmware might not support UEFI/MBR booting. So we create a GPT partition table in the USB stick image:
```
parted -s $USB_IMAGE_LOOPDEV mklabel gpt
```
We need to create a partition to contain UEFI-related files and to make that partition bootable; 50MB is adequate:
```
parted -s $USB_IMAGE_LOOPDEV mkpart fat32 0% 50M
parted -s $USB_IMAGE_LOOPDEV set 1 boot on
```
We need to create a partition to contain the Linux kernel, the initrd.gz file, which contains the Debian installer program, and the GRUB menus; 200MB is adequate:
```
parted -s $USB_IMAGE_LOOPDEV mkpart ext4 50M 100%
```
Note that as well as the files just mentioned, we will also need to put the answers to the questions that the Debian installer would otherwise ask somewhere. Precisely where it must go is discussed later.

Creating filesystems in the USB disk image

The EFI partition must be formatted with one of the following filesystems: FAT12, FAT16 or FAT32. (UEFI is the successor to EFI, but ‘EFI’ persists in the names of partitions and mountpoints.) The other partition can be formatted with any standard Linux filesystem, such as ext4.

Run:

mkfs -t vfat -F 32 -n EFIBOOT ${USB_IMAGE_LOOPDEV}p1
BOOT_UUID=$(uuidgen)
mkfs -t ext4 -U $BOOT_UUID ${USB_IMAGE_LOOPDEV}p2

Note that we generated a UUID, recorded it in a shell variable and then told mkfs to create a filesystem labelled with that UUID. This is important but why we do it is only explained later. For the time being, it is enough to know that we have created a filesystem and that we know its UUID.

A note about mountpoint relationships

Usually on an installed Linux OS, we have:

the Linux kernel and initrd.gz file and grub.cfg are stored in a filesystem that is mounted on /boot;
UEFI-related stuff is stored in a filesystem that is mounted on /boot/efi;
/ is a different filesystem to /boot and /boot/efi.

Hopefully those relationships are demonstrated by this I/O:

lagane# df -m /boot/efi /boot /
Filesystem       1M-blocks Used Available Use% Mounted on
/dev/vda1              523    6       518   1% /boot/efi
/dev/vda2              946   52       830   6% /boot
/dev/mapper/vg0-root 14354 8285      5322  61% /
lagane#

In preparing the USB stick image, we try to follow the same convention and have:

the kernel and initrd.gz file and grub.cfg are stored in a filesystem that is mounted on $USB_IMAGE_ROOTFS_MNTPNT/boot (we will define that variable in a minute);
UEFI-related stuff is stored in a filesystem that is mounted on $USB_IMAGE_ROOTFS_MNTPNT/boot/efi;
$USB_IMAGE_ROOTFS_MNTPNT is a different filesystem to $USB_IMAGE_ROOTFS_MNTPNT/boot and $USB_IMAGE_ROOTFS_MNTPNT/boot/efi.

Mounting the USB stick image’s filesystems (in order to put stuff in them)

We create a temporary directory as a container for the mountpoints for the two filesystems in the USB stick image and we record its location in a shell variable:
```
USB_IMAGE_ROOTFS_MNTPNT=$(mktemp -d)
```

Now we can create directories boot and boot/efi under that directory and mount the USB stick image’s filesystems on them:

mkdir $USB_IMAGE_ROOTFS_MNTPNT/boot
mount ${USB_IMAGE_LOOPDEV}p2 $USB_IMAGE_ROOTFS_MNTPNT/boot
mkdir $USB_IMAGE_ROOTFS_MNTPNT/boot/efi
mount ${USB_IMAGE_LOOPDEV}p1 $USB_IMAGE_ROOTFS_MNTPNT/boot/efi

Since we have not copied anything into the USB stick image yet, it is still basically empty:

lagane# find $USB_IMAGE_ROOTFS_MNTPNT/ 
/tmp/tmp.vNngrJdDCj/
/tmp/tmp.vNngrJdDCj/boot
/tmp/tmp.vNngrJdDCj/boot/efi
/tmp/tmp.vNngrJdDCj/boot/lost+found
lagane#

Adding UEFI-related files and making the USB stick image bootable

We run:

grub-install --removable --no-uefi-secure-boot --target=x86_64-efi \
    --efi-directory=$USB_IMAGE_ROOTFS_MNTPNT/boot/efi \
    --boot-directory=$USB_IMAGE_ROOTFS_MNTPNT/boot \
    --bootloader-id=grub --recheck $USB_IMAGE_LOOPDEV

(Hopefully the correlation between some of those option names and their values makes clear why we mounted ${USB_IMAGE_LOOPDEV}p2 on $USB_IMAGE_ROOTFS_MNTPNT/boot, rather than just mounting it directly on $USB_IMAGE_ROOTFS_MNTPNT.)

Verify that $USB_IMAGE_ROOTFS_MNTPNT now contains files under the boot/efi subdirectory, which grub-install just created:

lagane# find $USB_IMAGE_ROOTFS_MNTPNT/ 
/tmp/tmp.vNngrJdDCj/
/tmp/tmp.vNngrJdDCj/boot
/tmp/tmp.vNngrJdDCj/boot/efi
/tmp/tmp.vNngrJdDCj/boot/efi/EFI
/tmp/tmp.vNngrJdDCj/boot/efi/EFI/BOOT
/tmp/tmp.vNngrJdDCj/boot/efi/EFI/BOOT/BOOTX64.EFI
/tmp/tmp.vNngrJdDCj/boot/lost+found
/tmp/tmp.vNngrJdDCj/boot/grub
/tmp/tmp.vNngrJdDCj/boot/grub/fonts
/tmp/tmp.vNngrJdDCj/boot/grub/fonts/unicode.pf2
/tmp/tmp.vNngrJdDCj/boot/grub/grubenv
/tmp/tmp.vNngrJdDCj/boot/grub/x86_64-efi
/tmp/tmp.vNngrJdDCj/boot/grub/x86_64-efi/mpi.mod
/tmp/tmp.vNngrJdDCj/boot/grub/x86_64-efi/halt.mod
/tmp/tmp.vNngrJdDCj/boot/grub/x86_64-efi/macho.mod
<a-lot-more-.mod-files>
/tmp/tmp.vNngrJdDCj/boot/grub/locale
/tmp/tmp.vNngrJdDCj/boot/grub/locale/id.mo
/tmp/tmp.vNngrJdDCj/boot/grub/locale/sv.mo
/tmp/tmp.vNngrJdDCj/boot/grub/locale/pt_BR.mo
<a-lot-more-.mo-files>
lagane#

If we were to try to boot this USB stick image now then GRUB would load but it would not know how to proceed because grub.cfg is still missing and so would just await instructions from the user:

Adding the Linux kernel and initrd.gz

The USB stick itself runs Linux and contains the Debian installer program. So we download both and copy then into the USB stick image.

Run:

cd $USB_IMAGE_ROOTFS_MNTPNT/boot
wget http://deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux
wget http://deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
cd -

A note about glue

GRUB and the Linux kernel are both now installed. But we are still missing the glue that makes GRUB call Linux. This glue will be provided by grub.cfg (specifically by the linux directive) in the USB stick image, which we will create later in this procedure.

The Linux kernel and the initrd.gz file, which contains the Debian installer, are both now installed. But we are still missing the glue that makes the Linux kernel call the Debian installer. Again, this glue will be provided by grub.cfg (specifically by the initrd directive).

We are about to create a preseeding configuration file containing the answers to all the questions that the Debian installer will ask. But after that we will still be missing the glue that makes the Debian installer load preseed.cfg. Yet again, this glue will be provided by grub.cfg (specifically by passing the text preseed/file and the path of the configuration file to the Debian installer via the kernel command line, which is specified with the linux directive).

Preseeding basics

By convention the preseeding configuration file is called preseed.txt or preseed.cfg. However, as long as the filename and the reference to it agree, it can be called anything. We will use preseed.cfg, as this is the name of the official example configuration file.

preseed.cfg lists of a lot of assignment statements of the form:

<installer-subsystem> <variable> <type> <value>

where: <installer-subsystem> is almost always d-i; <type> is one of string, boolean, password, select, multiselect or note; <value> can be multiple words and should not be quoted – it extends to the end of line or beyond it if the line ends with a backslash.

During an installation, the Debian installer checks if a particular variable, whose value it needs, is set; if it is not then it asks the user; if it is set then it uses its value.

The settings made below are for my environment (e.g. I want a static IP configuration; I do not want swap space). For other configurations (e.g. you want a dynamic IP configuration; you want swap space), the best reference material is the official example preseed.cfg.

A word about generating preseed.cfg and grub.cfg

Imagine that I want to add this comment to preseed.cfg:

#  This file was created on Sunday 25 April 2021.

That is easy enough to understand but, unfortunately, because the name of the file to which I want to add that line appears in the text before the code block (i.e. in the sentence “imagine that I want to add this comment to preseed.cfg”) rather than in the code block, it is much less copy-and-paste-able.

For this reason I usually generate such lines using echo statements, shell variables and little bit of shell scripting:

PRESEEDCFG_FILE=preseed.cfg
TODAY=$(date '+%A %d %B %Y')


{
    echo "#  This file was created on $TODAY."
} >> $PRESEEDCFG_FILE

That is much more copy-and-paste-able but, unfortunately, it is also much less readable! So, in the sections below that are concerned with adding stuff to preseed.cfg and grub.cfg, I compromise and do both: first I show the line that I want to add – and if necessary explain it – and then later I run the code to actually do it for real.

Preseeding the hostname

I’m installing a host with fully qualified hostname questaroli.pasta.net. So I set the hostname and the domain with these assignments:

d-i netcfg/get_hostname string questaroli
d-i netcfg/get_domain string pasta.net

Preseeding network settings

I use a static network configuration. So I set the network parameters with these assignments:

d-i netcfg/disable_autoconfig boolean true
d-i netcfg/confirm_static boolean true
d-i netcfg/get_ipaddress string 192.168.1.12
d-i netcfg/get_netmask string 255.255.255.0
d-i netcfg/get_gateway string 192.168.1.51
d-i netcfg/get_nameservers string 192.168.1.21

Preseeding access to a package repository

I instruct the Debian installer to get packages from deb.debian.org; that particular server automatically redirects the Debian installer to a nearby mirror:

d-i mirror/country string manual
d-i mirror/http/hostname string deb.debian.org
d-i mirror/http/directory string /debian/
d-i mirror/http/proxy string

Note that the proxy assignment does not have a value: I do not want to or need to set a proxy but I do not want to be asked the question either.

Preseeding users and passwords

As mentioned above, I do not create any normal users; I have a site-specific script to do that, which I run after the basic OS is installed. So I just need to set root’s password and confirm that I do not want to make any normal users.

To specify root’s password there are two options: specify it in plaintext or specify it in encrypted text; we will do the latter. We could use the mkpasswd command to determine the encrypted password. For example, if the password was ‘mysecret’ then could we get the encrypted password by running:

lagane# mkpasswd mysecret
M2jkkhb60Oj1c
lagane#

and then we could put in preseed.cfg:

d-i passwd/root-password-crypted password M2jkkhb60Oj1c
d-i passwd/make-user boolean false

But, as mentioned above, we generate preseed.cfg and so we are able to combine those two blocks into one. See below for details.

Preseeding which disk to use

We need to tell the Debian installer the answers to two questions:

On which disk should it install the OS?
On which disk should it install GRUB?

Since the answers to these questions vary from system to system (e.g. a single disk system might see the first hard disk as /dev/sda but with a USB stick inserted the first hard disk might be /dev/sdb; a VM might use /dev/vda). So what we really need to tell the Debian installer is how to work out of the answers to these questions for itself.

We can provide a few hints:

The Debian installer should install the OS and GRUB onto the same disk.
That disk will be the first real hard disk.

Using the tools available and these hints we can write a small program to determine the first hard disk:

#  list all USB partitions; strip the partition number to
#  get USB disks, deduplicate, save results to a file.
list-devices usb-partition | sed 's/.$//' | sort -u > /tmp/usb-disks

# list all disks (including USB sticks), sort, save results
# to a file.
list-devices disk | sort > /tmp/all-disks

#  from the list of all disks, remove the USB disks, take
#  first result, save result in a variable.
INSTALL_DISK=$(fgrep -vxf /tmp/usb-disks /tmp/all-disks | head -1)

Then we can use debconf-set to save that value into the two preseeding variables from which the Debian installer will read the OS disk and the GRUB disk:

debconf-set partman-auto/disk $INSTALL_DISK
debconf-set grub-installer/bootdev $INSTALL_DISK

We need to pack all of that into a single preseeding assignment and ensure it gets evaluated at the right time (i.e. after loading disk controller drivers that might reveal additional disks). We can use partman/early_command for this:

d-i partman/early_command string \
    list-devices usb-partition | sed 's/.$//' | sort -u > /tmp/usb-disks; \
    list-devices disk | sort > /tmp/all-disks; \
    INSTALL_DISK=$(fgrep -vxf /tmp/usb-disks /tmp/all-disks | head -1); \
    debconf-set partman-auto/disk $INSTALL_DISK; \
    debconf-set grub-installer/bootdev $INSTALL_DISK

That line looks much worse than it really is; it has the same format as all the other lines; namely:

<installer-subsystem> <variable> <type> <value>

and its value is the code in the two boxes above with the comments stripped and using semicolon-plus-trailing-backslash syntax to spread one line over several lines for readability.

Preseeding disk partitioning

As recommended for UEFI booting, we use GPT partitioning:

d-i partman-efi/non_efi_system boolean true
d-i partman-partitioning/choose_label string gpt
d-i partman-partitioning/default_label string gpt

We will use a custom partitioning recipe as this provides the most flexibility. However, even with a custom partitioning recipe, we need to answer this question:

which we do with this:

d-i partman-auto/method string lvm

The partitioning scheme that I want is:

small primary partitions mounted on /boot/efi and /boot,
the rest of the disk allocated to LVM, specifically volume group (VG) ‘vg0’,
15GB ‘root’ logical volume (LV) mounted on /,
no swap (because my site-specific script will handle that later),
most of the VG unallocated.

I could not get the Debian installer to respect the size specified for the last LV (actually, there is only one LV in the partitioning scheme, but if there were more then it would be the last one whose specified size would not be respected); it always created it using all available space in the VG. My workaround was to add an extra LV:

all remaining space in VG for LV ‘remainder’ not to be mounted

and then, at the last possible moment, just before the Debian installer reboots, to delete that LV, leaving most of the VG unallocated again.

We define a recipe called ‘myscheme’:

d-i partman-auto/expert_recipe string myscheme :: \
    550 550 550 fat32 \
    $primary{ } \
    { efi } format{ } \
    . \
    1024 1024 1024 ext4 \
    $primary{ } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
     mountpoint{ /boot } \
     . \
    15360 15360 15360 ext4 \
    method{ lvm } \
    $lvmok{ } \
    lv_name{ root } \ 
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
    mountpoint{ / } \
    . \
    1 1000000000 1000000000 affs1 \
    method{ keep } \
    $lvmok{ } \
    lv_name{ remainder } \
    .

(Note to Alexis: the next time I do this procedure, I should note if 550, 1024 and 15360 result in that many MB or MiB; I want MiB but I suspect it will be MB, for the reasons described in this article.)

It is not enough just to define a recipe; it is also necessary to choose it:

d-i partman-auto/choose_recipe select myscheme

Note that we cannot specify the name of the VG in which the LVs are to be created; that we must do with another assignment:

d-i partman-auto-lvm/new_vg_name string vg0

Although I could not stop the last LV from consuming all the free space in the VG (and therefore needed to create that LV for /remainder), it is possible to stop the LVM partition from consuming all the free space on the disk. However, that is not what I want; I want that LVM partition to consume all the free space on the disk:

d-i partman-auto-lvm/guided_size string max

Regarding the removal of the /remainder LV at the last moment: there is a variable to define commands to be run just before the reboot:

d-i preseed/late_command string lvremove -f /dev/vg0/remainder

Preseeding package selection

As mentioned above, I want a minimal installation. Therefore I select only the ‘standard packages’ package group:

tasksel tasksel/first multiselect standard

Note that the installer subsystem is not d-i, but tasksel.

To specify individual packages to install it is possible to call apt-install (one of the helper commands) in the preseed/late_command assignment:

d-i preseed/late_command string lvremove -f /dev/vg0/remainder; apt-install zsh

However, I found that preseed/late_command was super-sensitive about formatting! This:

d-i preseed/late_command string lvremove -f /dev/vg0/remainder; \
    apt-install zsh; \
    apt-install something-else; \
    apt-install and-one-more-thing; \

(i.e. a list of semicolon-backslash-terminated lines plus a blank line to terminate the line splitting) caused the installer to ask:

Write changes to disk and configure LVM?

Of course, if one gets asked a question like that, then one thinks that preseeding disk partitioning went wrong, not that the late command might be wrong. So I install individual packages in my site-specific script, which I run after the installation is finished.

Preseeding miscellaneous confirmations

When performing a manual installation, confirmations are required (e.g. writing the partition table to disk; leaving the disk partitioning menu without having created a swap partition). When performing a preseeded installation, we are required to make the same confirmations:

#  Don't load non-free drivers at install time (has no effect if not needed)
d-i hw-detect/load_firmware boolean true
# Participate in package usage survey? No.
popularity-contest popularity-contest/participate boolean false
# Write the changes and configure LVM? Yes.
d-i partman-lvm/confirm boolean true
# Remove existing logical volume data? Yes.
d-i partman-lvm/device_remove_lvm boolean true
# Finish partitioning and write changes to disk? Yes.
d-i partman/choose_partition select finish
# No file system is specified for partition #1 of LVM VG vg0, LV remainder. Go back? No.
d-i partman-basicmethods/method_only boolean false
# You have not selected any partitions for use as swap space. Return to partitioning menu? No.
d-i partman-basicfilesystems/no_swap boolean false
# Write the changes to disk? Yes.
d-i partman/confirm boolean true
# The installation is complete, ... Continue!
d-i finish-install/reboot_in_progress note

Preseeding regional settings

Under the Debian installer’s advanced options boot menu there is the option to perform an automated install, which leads to this screen:

Note that it is asking this question in English! What this demonstrates is that specifying in the preseed.cfg that the Debian installer should display messages in another language is already too late! The language (and other regional settings) need to be made earlier in boot process. The fact that this screen is only reachable via the advanced options boot menu is not relevant. We have two options:

put regional settings in preseed.cfg and embed that file inside initrd.gz;
pass regional settings to the Debian installer by setting them on the kernel command line in grub.cfg.

We will take that second approach when we generate grub.cfg. For the time being, it is enough to know that the settings I will provide will be:

d-i debian-installer/language string en
d-i debian-installer/country string DE
d-i debian-installer/locale string en_GB.UTF-8
d-i keyboard-configuration/xkb-keymap string us

(That might look an odd combination: I’m a Brit living in Germany with a US international keyboard.)

Generating preseed.cfg

By running the pristine netinst USB stick image, letting it finish loading additional components, and then pressing ALT-F2 to flip to the second console terminal, we can explore the mounted filesystems that are available while the Debian installer is running:

From the above it should be clear: the EFI partition gets mounted on /media and, therefore, the best place to put preseed.cfg is /boot/efi/preseed.cfg inside the USB stick image, which will appear as /media/preseed.cfg when the USB stick image is actually booted.

To generate preseed.cfg we run:

#  configurable stuff
DI_NETWORK_HOST_NAME=questaroli
DI_NETWORK_HOST_DOMAIN=pasta.net
DI_NETWORK_HOST_IPADDR=192.168.1.12
DI_NETWORK_NET_MASK=255.255.255.0
DI_NETWORK_GATEWAY_IPADDR=192.168.1.51
DI_NETWORK_DNS_IPADDR=192.168.1.21
#  password is single quoted to best protect any funny chars in it
DI_USERS_ROOT_PASSWD='mysecret' 

#  non-configurable stuff
PRESEEDCFG_FILE=$USB_IMAGE_ROOTFS_MNTPNT/boot/efi/preseed.cfg
DI_USERS_ROOT_PASSWD_ENCRYPYTED="$(mkpasswd "$DI_USERS_ROOT_PASSWD")" 
{
    echo "#  regional settings"
    echo "#  (see grub.cfg)"
 
    echo "# hostname and and domain name"
    echo "d-i netcfg/get_hostname string $DI_NETWORK_HOST_NAME"
    echo "d-i netcfg/get_domain string $DI_NETWORK_HOST_DOMAIN"
    echo "#  network configuration"
    echo "d-i netcfg/disable_autoconfig boolean true"
    echo "d-i netcfg/get_ipaddress string $DI_NETWORK_HOST_IPADDR"
    echo "d-i netcfg/get_netmask string $DI_NETWORK_NET_MASK"
    echo "d-i netcfg/get_gateway string $DI_NETWORK_GATEWAY_IPADDR"
    echo "d-i netcfg/get_nameservers string $DI_NETWORK_DNS_IPADDR"
    echo "d-i netcfg/confirm_static boolean true"
    echo "#  access to a package repository"
    echo "d-i mirror/country string manual"
    echo "d-i mirror/http/hostname string deb.debian.org"
    echo "d-i mirror/http/directory string /debian/"
    echo "d-i mirror/http/proxy string"

    echo "#  users and passwords"
    echo "d-i passwd/root-password-crypted password $DI_USERS_ROOT_PASSWD_ENCRYPYTED"
    echo "d-i passwd/make-user boolean false"

    echo "#  which disk to install onto"
    echo "d-i partman/early_command string \\" 
    echo " list-devices usb-partition | sed 's/.$//' | sort -u > /tmp/usb-disks; \\" 
    echo " list-devices disk | sort > /tmp/all-disks; \\" 
    echo " INSTALL_DISK=\$(fgrep -vxf /tmp/usb-disks /tmp/all-disks | head -1); \\"
    echo " debconf-set partman-auto/disk \$INSTALL_DISK; \\" 
    echo " debconf-set grub-installer/bootdev \$INSTALL_DISK" 

    echo "#  GPT partitioning"
    echo "d-i partman-efi/non_efi_system boolean true"
    echo "d-i partman-partitioning/choose_label string gpt"
    echo "d-i partman-partitioning/default_label string gpt"

    echo "#  partitioning"
    echo "d-i partman-auto/method string lvm"
    echo "d-i partman-auto/expert_recipe string myscheme :: \\"
    echo " 550 550 550 fat32 \\" 
    echo " \$primary{ } \\"
    echo " method{ efi } format{ } \\"
    echo " . \\"
    echo " 1024 1024 1024 ext4 \\"
    echo " \$primary{ } \\"
    echo " method{ format } \\"
    echo " format{ } \\"
    echo " use_filesystem{ } \\"
    echo " filesystem{ ext4 } \\"
    echo " mountpoint{ /boot } \\"
    echo " . \\"
    echo " 15360 15360 15360 ext4 \\"
    echo " method{ lvm } \\"
    echo " \$lvmok{ } \\"
    echo " lv_name{ root } \\"
    echo " format{ } \\"
    echo " use_filesystem{ } \\"
    echo " filesystem{ ext4 } \\"
    echo " mountpoint{ / } \\"
    echo " . \\"
    echo " 1 1000000000 1000000000 affs1 \\"
    echo " method{ keep } \\"
    echo " \$lvmok{ } \\"
    echo " lv_name{ remainder } \\"
    echo " . \\"
    echo
    echo "d-i partman-auto/choose_recipe select myscheme"
    echo "d-i partman-auto-lvm/new_vg_name string vg0"
    echo "d-i partman-auto-lvm/guided_size string max"
    echo "d-i preseed/late_command string lvremove -f /dev/vg0/remainder"

    echo "#  packages"
    echo "tasksel tasksel/first multiselect standard"

    echo "#  miscellaneous confirmations"
    echo "#  Don't load non-free drivers at install time (has no"
    echo "#  effect if not needed)"
    echo "d-i hw-detect/load_firmware boolean true"
    echo "#  Participate in package usage survey? No."
    echo "popularity-contest popularity-contest/participate boolean false"
    echo "#  Write the changes and configure LVM? Yes. "
    echo "d-i partman-lvm/confirm boolean true"
    echo "#  Remove existing logical volume data? Yes."
    echo "d-i partman-lvm/device_remove_lvm boolean true"
    echo "#  Finish partitioning and write changes to disk? Yes."
    echo "d-i partman/choose_partition select finish"
    echo "#  No file system is specified for partition #1 of"
    echo "#  LVM VG vg0, LV remainder. Go back? No."
    echo "d-i partman-basicmethods/method_only boolean false"
    echo "#  You have not selected any partitions for use as"
    echo "#  swap space. Return to partitioning menu? No."
    echo "d-i partman-basicfilesystems/no_swap boolean false"
    echo "#  Write the changes to disk? Yes."
    echo "d-i partman/confirm boolean true"
    echo "#  The installation is complete, ... Continue!"
    echo "d-i finish-install/reboot_in_progress note"
} > $PRESEEDCFG_FILE

Note that while writing this article, I needed to examine the Debian installer environment while it was running. In case you need to do the same then: commenting out the assignment for netcfg/get_hostname will make the installer pause early in the installation; commenting out the assignment for finish-install/reboot_in_progress will make it pause late in the installation.

Gluing GRUB to Linux with grub.cfg

As mentioned above, grub.cfg provides the glue that makes GRUB call Linux. A minimal grub.cfg to do this is:

linux (hd0,1)/linux

In this example the ‘root device’ – that is the device containing the Linux kernel and the initrd.gz file – is specified using the GRUB-ish name (hd0,1). So GRUB would load the Linux kernel from the top directory on the second partition (partition #1) on the first disk-like device (disk #0).

But, at the time that we want to create grub.cfg, we cannot be sure that that those numbers are correct: some systems will put the USB stick ahead of the first hard disk, other systems will do it the other way round. We need to find a way to make GRUB work out the root device for itself.

As a first step, we could set the root device name in a variable and then reference that variable:

set root=(hd0,1)
linux /linux

The linux directive knows that if the root device is not specified in its parameter then it should consult the value of the root variable. Okay, that did not help much, but it will help us understand the next step.

A nicer alternative to set the root device would be to search for the root device; we could search using the UUID of the filesystem on the root device as the search criterion:

search --no-floppy --fs-uuid --set=root <uuid>

But to specify the UUID in the search directive in grub.cfg requires that we would need know the UUID in advance. We could do this by making the filesystem and then querying its UUID, but simpler would be to:

generate a random UUID,
save it in a variable,
reference it when making the filesystem,
reference it in the search directive.

If you scroll back up to ‘Creating filesystems in the USB disk image’, then you will see we already did the first three of these four steps! What is left to do is to write that search line with the value of $BOOT_UUID:

search --no-floppy --fs-uuid --set=root $BOOT_UUID
linux /linux

Actually, it is not possible to set BOOT_UUID in a shell session and then reference $BOOT_UUID from inside grub.cfg. But when we generate grub.cfg it will be possible.

Gluing Linux to the Debian installer with grub.cfg

Again, as mentioned above, grub.cfg provides the glue that makes the Linux kernel call the Debian installer.

Size restrictions mean that the Linux kernel cannot be built with all the drivers for all the hardware that it might encounter. Therefore many drivers are split off into a separate initrd.gz file – an initial RAM disk image. (This is gross simplification: there are several other important reasons why we use an initrd.gz file in the boot process.)

A normal initrd.gz file loads those additional drivers (e.g. loading disk controller drivers that might reveal additional disks), mounts the real root filesystem and boots the OS found there.

When doing a Debian installation, a special initrd.gz file loads those additional drivers, but skips mounting the real root filesystem (why bother mounting it when it is about to be reformatted?) and runs the Debian installer, which is contained within the initrd.gz file.

Loading the initrd.gz file into memory and telling the Linux kernel where it is is GRUB’s responsibility and we tell GRUB to do it by adding this to grub.cfg:

initrd /initrd.gz

Gluing the Debian installer to preseed.cfg with grub.cfg

Yet again, as mentioned above, grub.cfg provides the glue that makes the Debian installer load preseed.cfg.

To do this we get GRUB to append arbitrary strings to the kernel command line; the Debian installer knows to inspect the kernel command line for settings that are being passed to it in this way. The kernel command line is set with the linux directive:

linux ... preseed/file=/media/preseed.cfg

(The ... represents the stuff that I already mentioned is needed on the linux line.)

Preseeding regional settings in grub.cfg

As mentioned above, regional settings cannot easily be preseeded in preseed.cfg. An alternative is to put them on the kernel command line; the Debian installer knows to inspect the kernel command line for settings that are being passed to it in this way:

linux ... debian-installer/language=en debian-installer/country=DE debian-installer/locale=UTF-8.en_GB keyboard-configuration/xkb-keymap=us

Note that, unlike when making assignments in preseed.cfg, these assignments do not specify the installer subsystem.

Generating grub.cfg

As mentioned above, we need to generate grub.cfg with echo statements so that we can reference $BOOT_UUID.

Additionally, we take the opportunity to expand grub.cfg to:

encapsulate the configuration in a menu entry whose title has the date and time when grub.cfg was generated (this helps with debugging),
add a timeout so that the user has time to see the menu and abort the installation before it starts,
verify the integrity of preseed.cfg by specifying its checksum,
give GRUB some control over video hardware (without insmod all_video GRUB will complain error: no suitable video mode found.),
disable IPv6.

Note that the file must be called grub.cfg and must be in /boot/grub (inside the USB stick image), otherwise GRUB will not load it.

To generate grub.cfg we run:

#  configurable stuff
DI_REGIONAL_LANGUAGE=en
DI_REGIONAL_COUNTRY=DE
DI_REGIONAL_LOCALE=en_GB.UTF-8
DI_REGIONAL_KEYMAP=us

#  non-configurable stuff
GRUBCFG_FILE=$USB_IMAGE_ROOTFS_MNTPNT/boot/grub/grub.cfg
PRESEEDCFG_MD5SUM=$(md5sum $PRESEEDCFG_FILE | awk '{ print $1 }')
TIMESTAMP=$(date +%Y%m%d%H%M%S)

{
    echo "set timeout=10"
    echo "menuentry \"Debian installer ($TIMESTAMP)\" {"
    echo "    search --no-floppy --fs-uuid --set=root $BOOT_UUID"
    echo "    insmod all_video"
    echo -n "    linux /linux"
    echo -n " debian-installer/language=$DI_REGIONAL_LANGUAGE"
    echo -n " debian-installer/country=$DI_REGIONAL_COUNTRY"
    echo -n " debian-installer/locale=$DI_REGIONAL_LOCALE"
    echo -n " keyboard-configuration/xkb-keymap=$DI_REGIONAL_KEYMAP"
    echo -n " preseed/file=/media/preseed.cfg"
    echo -n " preseed/file/checksum=$PRESEEDCFG_MD5SUM"
    echo -n " ipv6.disable=1"
    echo
    echo "    initrd /initrd.gz"
    echo "}"
} > $GRUBCFG_FILE

Note that if you want to put preseed.cfg on a webserver and specify its URL in grub.cfg, beware that auto should be added to the kernel command line; the effect of that is to make the Debian installer postpone the configuration of hostname, locale and keymap so that they can be answered from preseed.cfg that got loaded from the network.

Note that we have now written everything to the partitions and they are nowhere near full:

lagane# df -h /dev/loop0p1 /dev/loop0p2
Filesystem   Size Used Avail Use% Mounted on
/dev/loop0p1  37M 141K   37M   1% /tmp/mkdebusbimg.LSikPM/boot/efi
/dev/loop0p2 191M  46M  132M  26% /tmp/mkdebusbimg.LSikPM/boot
lagane#

But reducing the USB stick image size or the size of the two partitions much below the values used above triggers these errors:

Warning: The resulting partition is not properly aligned for best performance.
Warning: The resulting partition is not properly aligned for best performance.
WARNING: Not enough clusters for a 32 bit FAT!

(The first two identical warnings appear during partitioning and the third warning appears while formatting the EFI partition.)

Tidying up

We unmount the USB stick image’s partitions and release the loop device:

umount $USB_IMAGE_ROOTFS_MNTPNT/boot/efi
umount $USB_IMAGE_ROOTFS_MNTPNT/boot
losetup -d $USB_IMAGE_LOOPDEV

We remove the temporary mountpoints:

rmdir $USB_IMAGE_ROOTFS_MNTPNT/boot
rmdir $USB_IMAGE_ROOTFS_MNTPNT

A script to do it all

Above we used shell variables and echo statements to generate preseed.cfg and grub.cfg. As you might have guessed, this means that converting this procedure into a shell script is easy. Here is the script. It is all the code above plus:

a GPL copyright banner explaining the script comes with absolutely no warranty!
a few sanity checks (e.g. are you root? are all the shell variables set?),
some messages to indicate what it is doing (and a few tiny functions to display these messages),
inclusion of the name of the script in the names of temporary files and directories.

You should absolutely not run it without first reviewing it and customise it to suit your own requirements; a few shell variables are deliberately not set in order to force you to customise them. If you run it without customising variables it will do this:

lagane# ./mkdebusbimg 
mkdebusbimg: INFO: doing some sanity checks ...
mkdebusbimg: ERROR: DI_NETWORK_HOST_NAME, DI_NETWORK_HOST_DOMAIN DI_NETWORK_HOST_IPADDR DI_NETWORK_NET_MASK DI_NETWORK_GATEWAY_IPADDR DI_NETWORK_DNS_IPADDR DI_USERS_ROOT_PASSWD: these variables are not set (hint: edit this script and make sure they are all set near the top)
lagane#

Once you have customised it it should to this:

lagane# ./mkdebusbimg 
mkdebusbimg: INFO: doing some sanity checks ...
mkdebusbimg: INFO: creating empty disk image ...
mkdebusbimg: INFO: detemining loop device to use ...
mkdebusbimg: INFO: setting up loop device ...
mkdebusbimg: INFO: partitioning ...
mkdebusbimg: INFO: making filesystems ...
mkdebusbimg: INFO: mounting file systems ...
mkdebusbimg: INFO: installing grub ...
mkdebusbimg: INFO: downloading Linux kernel and initrd.gz ...
mkdebusbimg: INFO: preparing /tmp/mkdebusbimg.T8nz0J/boot/efi/preseed.cfg ...
mkdebusbimg: INFO: preparing /tmp/mkdebusbimg.T8nz0J/boot/grub/grub.cfg ...
mkdebusbimg: INFO: tidying up ...
mkdebusbimg: INFO: USB stick image is /tmp/mkdebusbimg.img
lagane#

Copying the USB stick image onto a USB stick

This is already well documented.