Cyber Services / All lessons / CentOS / Linux Administration / Controlling the Boot Process in RHEL 8

Controlling the Boot Process in RHEL 8

Module 12 of 33 · 13 min read · Free to read

MODULES ↓ Command blocks have a COPY button

Master systemd Targets, Rescue Mode, Emergency Mode, Root Password Recovery, and File System Repair

Understanding and controlling the boot process is essential for Linux system administrators. RHEL 8 uses systemd as its init system, which provides a structured approach to system initialization through targets. When things go wrong — forgotten passwords, corrupted file systems, or boot failures — knowing how to intervene at various boot stages can save your system from disaster.

This comprehensive guide covers the complete RHEL 8 boot process, systemd targets, rescue and emergency modes, root password recovery, and file system repair techniques. All procedures are tested and verified on RHEL 8.5.

The RHEL 8 Boot Process: A Deep Dive

The System Boot Analogy

Think of system boot as a rocket launch. The BIOS/UEFI is the launch control checking all systems. The bootloader (GRUB2) is the countdown sequence that prepares for ignition. The kernel is the rocket engine firing up. Finally, systemd is the mission control that coordinates all subsystems (targets) to achieve orbit (fully operational system).

Boot Process Phases

Phase 1: Hardware Initialization (BIOS/UEFI)

What happens: Power-on self-test (POST), hardware detection, firmware initialization

Key files: None (firmware level)

Intervention point: BIOS/UEFI setup, boot device selection

Phase 2: Bootloader Stage (GRUB2)

What happens: GRUB2 loads from /boot/grub2/, presents boot menu, loads kernel and initramfs

Key files: /boot/grub2/grub.cfg, /boot/vmlinuz-*, /boot/initramfs-*.img

Intervention point: GRUB2 menu editing (press 'e' during boot)

Phase 3: Kernel Initialization

What happens: Kernel decompresses, initializes hardware, mounts root filesystem (via initramfs if needed)

Key files: Kernel modules in /lib/modules/, device files in /dev/

Intervention point: Kernel command line parameters, early debug shell

Phase 4: systemd Initialization

What happens: systemd starts as PID 1, reads default target, activates services and dependencies

Key files: /etc/systemd/system/default.target, unit files in /usr/lib/systemd/system/

Intervention point: Target selection, rescue/emergency mode

Boot Process Timeline

  1. Power applied → BIOS/UEFI firmware executes
  2. BIOS/UEFI loads GRUB2 bootloader from MBR or EFI System Partition
  3. GRUB2 reads grub.cfg and presents boot menu
  4. User selects kernel (or default selection after timeout)
  5. GRUB2 loads kernel image and initramfs into memory
  6. Kernel starts, uses initramfs to mount root filesystem
  7. Kernel executes /sbin/init (which is systemd)
  8. systemd reads default target and starts required services
  9. System reaches desired target state (graphical, multi-user, etc.)

Understanding systemd Targets

systemd targets replace the traditional SysV runlevels and provide a more flexible way to define system states. Targets are units that represent synchronization points during boot or shutdown.

Common systemd Targets

poweroff.target

Equivalent to: Runlevel 0

Description: Halts the system completely

Command: systemctl poweroff

rescue.target

Equivalent to: Runlevel 1 (Single-user mode)

Description: Minimal system with root shell, basic services only

Command: systemctl rescue

emergency.target

Equivalent to: Runlevel 1 (Emergency mode)

Description: Minimal system with root shell, no services started

Command: systemctl emergency

multi-user.target

Equivalent to: Runlevel 3

Description: Multi-user text mode, network enabled, no GUI

Command: systemctl isolate multi-user.target

graphical.target

Equivalent to: Runlevel 5

Description: Multi-user graphical mode with desktop environment

Command: systemctl isolate graphical.target

reboot.target

Equivalent to: Runlevel 6

Description: Reboots the system

Command: systemctl reboot

Managing Targets

  1. 1 Check Current Default Target
    systemctl get-default

    This shows what target the system boots into by default.

  2. 2 List All Available Targets
    systemctl list-units --type=target
  3. 3 Change Default Target
    sudo systemctl set-default multi-user.target

    This changes the default boot target permanently.

  4. 4 Switch to Different Target (Temporary)
    sudo systemctl isolate graphical.target

    This switches to the specified target without changing the default.

  5. 5 View Target Dependencies
    systemctl list-dependencies multi-user.target

    Shows all services and units that must be active for the target.

Practice Lab 1: Target Management

Determine your current default target and change it temporarily:

# Check current default target
systemctl get-default

# List all active targets
systemctl list-units --type=target --state=active

# Switch to multi-user mode (text mode)
sudo systemctl isolate multi-user.target

# Verify you're in multi-user mode
systemctl list-units --type=target --state=active

# Switch back to graphical mode
sudo systemctl isolate graphical.target

# Set multi-user as permanent default (optional)
sudo systemctl set-default multi-user.target
OperationTraditional Commandsystemd Command
Reboot System reboot systemctl reboot
Power Off poweroff systemctl poweroff
Halt System halt systemctl halt
Suspend pm-suspend systemctl suspend
Hibernate pm-hibernate systemctl hibernate

Booting into Rescue Mode

Rescue mode provides a minimal environment with basic system services running, allowing you to repair your system when it won't boot normally. It's particularly useful for fixing configuration issues, recovering data, or resetting passwords.

Methods to Enter Rescue Mode

Method 1: From GRUB2 Boot Menu (Recommended)

  1. 1 Interrupt GRUB2 Boot Menu

    When the GRUB2 menu appears, press any key to stop the automatic boot countdown.

  2. 2 Edit Boot Entry

    Highlight the desired kernel entry and press 'e' to edit.

  3. 3 Modify Kernel Parameters

    Find the line starting with linux or linuxefi. Move to the end of this line and add:

    systemd.unit=rescue.target

    Alternatively, you can use the shorter form:

    rd.break

    The rd.break method breaks into the initramfs shell before the root filesystem is mounted.

  4. 4 Boot with Modified Parameters

    Press Ctrl+X or F10 to boot with the modified parameters.

Method 2: Using systemctl (From Running System)

sudo systemctl rescue

This will immediately switch to rescue mode, terminating all non-essential services and providing a root shell.

Practice Lab 2: GRUB2 Rescue Mode

Boot your system into rescue mode using GRUB2:

  1. Reboot your system
  2. When GRUB2 menu appears, press any key to stop countdown
  3. Press 'e' to edit the default boot entry
  4. Find the line starting with linux
  5. Add systemd.unit=rescue.target at the end of that line
  6. Press Ctrl+X to boot
  7. You'll be prompted for the root password (if set)
  8. Once in rescue mode, explore the minimal environment:
# Check mounted filesystems
mount | grep -v tmpfs

# Check available disk space
df -h

# List running processes
ps aux

# Exit rescue mode to continue normal boot
exit
Pro Tip: If you can't remember the exact kernel parameter, you can also use single or 1 which are aliases for rescue mode in RHEL 8.

Resetting the Root Password

Forgotten root passwords are a common administrative challenge. RHEL 8 provides secure methods to reset the root password using rescue mode or the initramfs shell.

Method 1: Using rd.break (Recommended for RHEL 8)

This method is preferred because it works even if the root account is locked or if SELinux is enforcing.

  1. 1 Boot with rd.break Parameter

    From GRUB2 menu, edit the kernel line and add rd.break at the end.

  2. 2 Access initramfs Shell

    After booting, you'll get a shell prompt in the initramfs environment:

    switch_root:/#
  3. 3 Remount Root Filesystem as Read-Write
    mount -o remount,rw /sysroot
    chroot /sysroot

    The chroot command changes the root directory to your actual system.

  4. 4 Reset Root Password
    passwd root

    Enter the new password twice when prompted.

  5. 5 Handle SELinux Context (Critical!)
    touch /.autorelabel

    This creates a file that tells SELinux to relabel all files on next boot. Without this, your system may fail to boot properly due to incorrect security contexts.

  6. 6 Exit and Continue Boot
    exit
    exit

    The system will now boot normally and automatically relabel files.

Method 2: Using rescue.target

This method works if you can boot into rescue mode and know the root password for rescue mode.

  1. Boot with systemd.unit=rescue.target
  2. Enter root password when prompted
  3. Run passwd root to change the password
  4. Type exit to continue normal boot

⚠️ Critical Security Note

The rd.break method requires physical or console access to the system. In production environments, ensure your BIOS/UEFI and GRUB2 are password-protected to prevent unauthorized password resets.

Practice Lab 3: Root Password Recovery

Practice resetting the root password in a test environment:

# From GRUB2 menu, add 'rd.break' to kernel line
# After getting initramfs shell:

mount -o remount,rw /sysroot
chroot /sysroot
passwd root
# Enter new password: MyNewSecurePassword123!
# Confirm password: MyNewSecurePassword123!
touch /.autorelabel
exit
exit

# System will boot and relabel files (takes several minutes)
# Login with new root password

Emergency Mode and Early Debug Shell

Emergency mode provides an even more minimal environment than rescue mode, with only the root filesystem mounted and no other services running. The early debug shell allows you to troubleshoot issues during the initramfs stage.

Emergency Mode

Emergency mode is useful when rescue mode fails to start or when you need absolute minimal system state.

  1. 1 Boot into Emergency Mode

    From GRUB2 menu, edit kernel line and add:

    systemd.unit=emergency.target
  2. 2 Root Password Required

    You'll be prompted for the root password. If you don't know it, you'll need to reset it first using the rd.break method.

  3. 3 Minimal Environment

    In emergency mode, only the root filesystem is mounted. No network, no other filesystems, no services.

Early Debug Shell

The early debug shell provides access during the initramfs stage, before the root filesystem is mounted. This is crucial for debugging storage or filesystem issues.

  1. 1 Enable Early Debug Shell

    From GRUB2 menu, edit kernel line and add:

    rd.debug

    This enables verbose debugging output and provides a shell if the boot process fails.

  2. 2 Alternative: Force Debug Shell

    To always get a debug shell during initramfs:

    rd.break rd.debug
  3. 3 Debug Shell Environment

    In the debug shell, you have access to basic commands like ls, cat, blkid, mount, etc.

    # List available block devices
    ls /dev/[sv]d*
    
    # Check filesystem types
    blkid
    
    # View boot logs
    cat /run/initramfs/rdsosreport.txt
    
    # Manually mount root filesystem
    mount /dev/sda2 /mnt/sysroot

Advanced Lab: Debugging Boot Failures

Simulate and debug a boot failure scenario:

# From GRUB2 menu, add 'rd.debug rd.break' to kernel line
# After getting debug shell:

# Check if root device exists
ls -l /dev/disk/by-uuid/

# Verify root= parameter matches actual UUID
cat /proc/cmdline

# Check if filesystem is corrupted
fsck -n /dev/sda2

# If fsck shows errors, repair filesystem (see next section)
# Manually mount root to verify contents
mount /dev/sda2 /mnt/sysroot
ls /mnt/sysroot/etc/fstab

# Exit debug shell to continue boot
exit

Repairing File System Issues at Boot

File system corruption is one of the most common causes of boot failures. RHEL 8 provides several methods to repair filesystem issues, both automatically and manually.

Automatic File System Check (fsck)

RHEL 8 automatically runs fsck during boot under certain conditions:

  • Filesystem was not cleanly unmounted (dirty bit set)
  • Maximum mount count reached (default: -1, disabled)
  • Maximum time between checks reached (default: 180 days)

Manual File System Repair

When automatic fsck fails or you need to force a check, manual intervention is required.

  1. 1 Boot into Rescue Mode

    Use rd.break or systemd.unit=rescue.target to get a shell.

  2. 2 Identify Problematic Filesystem
    # List all block devices
    lsblk
    
    # Check filesystem types and UUIDs
    blkid
    
    # View /etc/fstab from your system
    cat /sysroot/etc/fstab
  3. 3 Unmount the Filesystem
    # If in rescue mode, filesystems may be auto-mounted
    umount /dev/sda2

    Filesystems must be unmounted before running fsck.

  4. 4 Run File System Check
    # Check filesystem without making changes
    fsck -n /dev/sda2
    
    # Repair filesystem interactively
    fsck -y /dev/sda2
    
    # For XFS filesystems (common in RHEL 8)
    xfs_repair /dev/sda2

    Filesystem-Specific Tools

    • ext4: fsck.ext4, e2fsck
    • XFS: xfs_repair, xfs_check
    • Btrfs: btrfs check
  5. 5 Verify Repair Success
    # Mount filesystem to verify
    mount /dev/sda2 /mnt
    ls /mnt
    umount /mnt
  6. 6 Continue Normal Boot
    exit  # From rescue mode
    # or
    exit
    exit  # From rd.break mode

Practice Lab 4: File System Repair

Practice repairing a simulated filesystem corruption:

# WARNING: Only perform on test systems!
# First, create a test filesystem
dd if=/dev/zero of=/tmp/testfs.img bs=1M count=100
mkfs.ext4 /tmp/testfs.img

# Mount and create some files
mkdir /tmp/testmount
mount -o loop /tmp/testfs.img /tmp/testmount
echo "test data" > /tmp/testmount/testfile.txt
umount /tmp/testmount

# Simulate corruption by editing image directly
hexdump -C /tmp/testfs.img | head -20
# (In real scenario, corruption would be accidental)

# Run fsck to repair
fsck -y /tmp/testfs.img

# Verify repair
mount -o loop /tmp/testfs.img /tmp/testmount
cat /tmp/testmount/testfile.txt
umount /tmp/testmount

Real-World Scenario: Production Server Recovery

Your production web server won't boot due to filesystem corruption:

# Boot with rd.break
# Remount root as read-write
mount -o remount,rw /sysroot

# Check which filesystems are problematic
cat /sysroot/etc/fstab
blkid

# Unmount all filesystems except root
umount /sysroot/boot
umount /sysroot/home

# Run fsck on root filesystem
fsck -y /dev/sda2

# Run fsck on other filesystems
fsck -y /dev/sda1  # /boot
fsck -y /dev/sda3  # /home

# If using LVM, check volume groups
vgscan
vgchange -ay

# Verify all filesystems mount correctly
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
mount /dev/sda3 /mnt/home
ls /mnt/var/log
umount /mnt/{boot,home,}

# Continue boot
touch /sysroot/.autorelabel
exit
exit

Advanced Boot Troubleshooting

Common Boot Issues and Solutions

Issue: "No Root Device Found"

Cause: Kernel can't find the root filesystem specified in boot parameters.

Solution:

# Boot with rd.break
# Check actual root device UUID
blkid | grep -i root

# Compare with boot parameter
cat /proc/cmdline | grep root=

# If UUID changed, update /etc/fstab and GRUB configuration
# Or boot with correct root= parameter temporarily

Issue: "Kernel Panic - Not Syncing"

Cause: Critical kernel error, often due to missing modules or hardware issues.

Solution:

# Boot with older kernel from GRUB menu
# Or boot with rd.debug to see detailed error messages
# Check hardware (RAM test, disk health)
# Reinstall kernel packages if corrupted

Issue: "Timed Out Waiting for Device"

Cause: systemd can't find a device specified in /etc/fstab.

Solution:

# Boot into rescue mode
# Check /etc/fstab for incorrect device entries
# Comment out problematic lines temporarily
# Or update device UUIDs to match current system
blkid
vi /etc/fstab

GRUB2 Recovery

If GRUB2 itself is corrupted, you can reinstall it:

# From rescue mode or live CD
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot  # if separate /boot

# Chroot into system
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
chroot /mnt

# Reinstall GRUB2
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg

# Exit chroot and reboot
exit
reboot

Best Practices and Security Considerations

Security Best Practices

  • Password-protect GRUB2 - Prevent unauthorized boot parameter changes
  • Secure BIOS/UEFI - Set admin password and disable boot from external devices
  • Regular backups - Maintain system backups to recover from catastrophic failures
  • Document procedures - Keep recovery procedures documented and tested
  • Limit physical access - Control who can access system console

Operational Best Practices

  • Test recovery procedures - Regularly practice boot recovery in test environments
  • Monitor system logs for early warning signs of filesystem issues
  • Keep system updated with latest kernel and systemd packages
  • Maintain multiple kernel versions in GRUB2 menu
  • Use LVM for flexible storage management and snapshots

⚠️ Critical Reminders

  • Always create /.autorelabel after password reset in RHEL 8
  • Never run fsck on mounted filesystems
  • Backup /etc/fstab before making changes
  • Test GRUB2 configuration changes before relying on them
  • Document your system's specific recovery procedures

Quick Reference & Troubleshooting

🔧 Essential Boot Commands

# Check current target
systemctl get-default

# Change default target
systemctl set-default multi-user.target

# Switch to rescue mode
systemctl rescue

# Reboot system
systemctl reboot

# Power off system  
systemctl poweroff

# List all targets
systemctl list-units --type=target

# Check filesystem (unmounted)
fsck -y /dev/sda2

# Repair XFS filesystem
xfs_repair /dev/sda2

# View boot logs
journalctl -b

# View previous boot logs
journalctl -b -1

📌 GRUB2 Boot Parameters Reference

  • systemd.unit=rescue.target - Boot into rescue mode
  • systemd.unit=emergency.target - Boot into emergency mode
  • rd.break - Break into initramfs shell
  • rd.debug - Enable debug output and shell on failure
  • single or 1 - Alias for rescue mode
  • 3 - Boot into multi-user mode
  • 5 - Boot into graphical mode
  • quiet - Suppress most boot messages
  • rhgb - Red Hat Graphical Boot (splash screen)

📚 Additional Resources

  • man systemd.special - Documentation for special systemd targets
  • man grub - GRUB bootloader documentation
  • man fsck - File system check documentation
  • man xfs_repair - XFS repair documentation
  • /usr/share/doc/systemd/ - systemd documentation
  • journalctl -b - View current boot logs
Modules (33) ← CentOS / Linux Administration