From 00a5ccd35bdf40389878cb9b132bc4d2dc2683f1 Mon Sep 17 00:00:00 2001 From: "M. Vefa Bicakci" Date: Mon, 4 Dec 2023 16:52:53 +0000 Subject: [PATCH] GRUB configuration: Increase UEFI watchdog timeout This commit increases the UEFI watchdog timeout utilized by GRUB in StarlingX from 3 minutes to 20 minutes to prevent undesirable and arguably premature UEFI watchdog timeout-triggered reboots during the installation of StarlingX ISO images via BMC/iLO/iDRAC/platform-provided virtual media redirection features in conjunction with ISO images hosted on web servers. In more detail, a user reported that a StarlingX-based distribution's ISO image would not successfully install with platform-provided ISO image redirection when the ISO image in question was hosted on a web server, despite the bandwidth and latency between the platform network interface and the web server being acceptable. The same user reported that removing the "efi-watchdog enable ..." line from the GRUB configuration resolved the issue. The same issue was later reproduced locally with an HPE DL360g10 server, where the OAM network interface was able to download an ISO image from a local server on a different subnet at a rate of about 76 MiB/s. (While the OAM and the iLO network interfaces are likely not the same, we do not envision the network conditions to be vastly different when the two network paths are compared.) In our reproduction of the issue, the downloading of the kernel and the initramfs images takes approximately nine minutes and ten seconds, after which the "Linux version" banner is printed out by the kernel on the serial console, regardless of whether the "Enhanced Download Performance" setting is enabled in the iLO settings or not. Based on these experimental results, this commit changes the UEFI watchdog timeout from 3 minutes to a duration that is approximately two times the initial kernel/initramfs load time of 9 minutes and 10 seconds encountered in our experiments: 20 minutes. Note that this commit does not affect the GRUB configuration files that are used after installation. The timeout remains 3 minutes in "/boot/efi/EFI/BOOT/grub.cfg" on installed systems after this commit, which is appropriate as the GRUB configuration file in question is utilized for booting up from local storage (i.e., SSD or HDD). Verification: * The reported issue was confirmed by placing a StarlingX-based distribution's nightly build ISO image on a web server, and the iLO (out-of-band platform management firmware) of the HPE DL360g10 server under test was configured to boot up from the ISO image on the web server via virtual media redirection using an HTTP URL. The 3 minute UEFI watchdog timeout set by GRUB was observed to be insufficient and the server was seen to autonomously reboot in the middle of the loading of the kernel and/or initramfs images. * A custom ISO image was built with this commit. * The built ISO image was uploaded to the same web server and the iLO configuration was modified to boot up from the custom-built ISO image instead, also via an HTTP URL. The server was observed to load the kernel/initramfs and transfer the control to the Linux kernel in about 9 minutes and 10 seconds, regardless of the "Enhanced Download Performance" setting in the iLO. * The installation was allowed to continue. Without the "Enhanced Download Performance" setting, the installation finished in ~36 hours, whereas with the setting in question enabled, the installation finished in ~2 hours. We also observed that this setting did not affect the initial loading of the kernel and initramfs images by GRUB. Closes-Bug: 2046182 Change-Id: Iaadf304fcc1969350e399fcd89a06ce1102df223 Signed-off-by: M. Vefa Bicakci --- .../config/debian/common/base-bullseye.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/debian-mirror-tools/config/debian/common/base-bullseye.yaml b/debian-mirror-tools/config/debian/common/base-bullseye.yaml index 55511175..0dbf31f2 100644 --- a/debian-mirror-tools/config/debian/common/base-bullseye.yaml +++ b/debian-mirror-tools/config/debian/common/base-bullseye.yaml @@ -110,13 +110,13 @@ iso-grub-entry: | submenu 'UEFI Debian Controller Install' --unrestricted --id=standard { menuentry 'Serial Console' --unrestricted --id=serial { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-std %BOOT_PARAMS% traits=controller defaultkernel=vmlinuz-*[!t]-amd64 initrd @INITRD@ } menuentry 'Graphical Console' --unrestricted --id=graphical { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-std %BOOT_PARAMS% traits=controller defaultkernel=vmlinuz-*[!t]-amd64 console=tty1 initrd @INITRD@ } @@ -125,13 +125,13 @@ iso-grub-entry: | submenu 'UEFI Debian All-in-one Install' --unrestricted --id=aio { menuentry 'Serial Console' --unrestricted --id=serial { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-std %BOOT_PARAMS% traits=controller,worker defaultkernel=vmlinuz-*[!t]-amd64 initrd @INITRD@ } menuentry 'Graphical Console' --unrestricted --id=graphical { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-std %BOOT_PARAMS% traits=controller,worker defaultkernel=vmlinuz-*[!t]-amd64 console=tty1 initrd @INITRD@ } @@ -140,13 +140,13 @@ iso-grub-entry: | submenu 'UEFI Debian All-in-one (lowlatency) Install' --unrestricted --id=aio-lowlat { menuentry 'Serial Console' --unrestricted --id=serial { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-rt %BOOT_PARAMS% traits=controller,worker,lowlatency defaultkernel=vmlinuz-*-rt-amd64 efi=runtime initrd @INITRD@ } menuentry 'Graphical Console' --unrestricted --id=graphical { set fallback=1 - efi-watchdog enable 0 180 + efi-watchdog enable 0 1200 linux /bzImage-rt %BOOT_PARAMS% traits=controller,worker,lowlatency defaultkernel=vmlinuz-*-rt-amd64 efi=runtime console=tty1 initrd @INITRD@ }