Master systemd Targets, Rescue Mode, Emergency Mode, Root Password Recovery, and File System Repair
Understanding and controlling the boot process is essential for Linux system administrators. RHEL 8 uses systemd as its init system, which provides a structured approach to system initialization through targets. When things go wrong — forgotten passwords, corrupted file systems, or boot failures — knowing how to intervene at various boot stages can save your system from disaster.
This comprehensive guide covers the complete RHEL 8 boot process, systemd targets, rescue and emergency modes, root password recovery, and file system repair techniques. All procedures are tested and verified on RHEL 8.5.
The RHEL 8 Boot Process: A Deep Dive
The System Boot Analogy
Think of system boot as a rocket launch. The BIOS/UEFI is the launch control checking all systems. The bootloader (GRUB2) is the countdown sequence that prepares for ignition. The kernel is the rocket engine firing up. Finally, systemd is the mission control that coordinates all subsystems (targets) to achieve orbit (fully operational system).
Boot Process Phases
Phase 1: Hardware Initialization (BIOS/UEFI)
What happens: Power-on self-test (POST), hardware detection, firmware initialization
Key files: None (firmware level)
Intervention point: BIOS/UEFI setup, boot device selection
Phase 2: Bootloader Stage (GRUB2)
What happens: GRUB2 loads from /boot/grub2/, presents boot menu, loads kernel and initramfs
Key files: /boot/grub2/grub.cfg, /boot/vmlinuz-*, /boot/initramfs-*.img
Intervention point: GRUB2 menu editing (press 'e' during boot)
Phase 3: Kernel Initialization
What happens: Kernel decompresses, initializes hardware, mounts root filesystem (via initramfs if needed)
Key files: Kernel modules in /lib/modules/, device files in /dev/
Intervention point: Kernel command line parameters, early debug shell
Phase 4: systemd Initialization
What happens: systemd starts as PID 1, reads default target, activates services and dependencies
Key files: /etc/systemd/system/default.target, unit files in /usr/lib/systemd/system/
Intervention point: Target selection, rescue/emergency mode
Boot Process Timeline
- Power applied → BIOS/UEFI firmware executes
- BIOS/UEFI loads GRUB2 bootloader from MBR or EFI System Partition
- GRUB2 reads
grub.cfgand presents boot menu - User selects kernel (or default selection after timeout)
- GRUB2 loads kernel image and initramfs into memory
- Kernel starts, uses initramfs to mount root filesystem
- Kernel executes
/sbin/init(which is systemd) - systemd reads default target and starts required services
- System reaches desired target state (graphical, multi-user, etc.)
Understanding systemd Targets
systemd targets replace the traditional SysV runlevels and provide a more flexible way to define system states. Targets are units that represent synchronization points during boot or shutdown.
Common systemd Targets
poweroff.target
Equivalent to: Runlevel 0
Description: Halts the system completely
Command: systemctl poweroff
rescue.target
Equivalent to: Runlevel 1 (Single-user mode)
Description: Minimal system with root shell, basic services only
Command: systemctl rescue
emergency.target
Equivalent to: Runlevel 1 (Emergency mode)
Description: Minimal system with root shell, no services started
Command: systemctl emergency
multi-user.target
Equivalent to: Runlevel 3
Description: Multi-user text mode, network enabled, no GUI
Command: systemctl isolate multi-user.target
graphical.target
Equivalent to: Runlevel 5
Description: Multi-user graphical mode with desktop environment
Command: systemctl isolate graphical.target
reboot.target
Equivalent to: Runlevel 6
Description: Reboots the system
Command: systemctl reboot
Managing Targets
- 1 Check Current Default Target
systemctl get-defaultThis shows what target the system boots into by default.
- 2 List All Available Targets
systemctl list-units --type=target - 3 Change Default Target
sudo systemctl set-default multi-user.targetThis changes the default boot target permanently.
- 4 Switch to Different Target (Temporary)
sudo systemctl isolate graphical.targetThis switches to the specified target without changing the default.
- 5 View Target Dependencies
systemctl list-dependencies multi-user.targetShows all services and units that must be active for the target.
Practice Lab 1: Target Management
Determine your current default target and change it temporarily:
# Check current default target
systemctl get-default
# List all active targets
systemctl list-units --type=target --state=active
# Switch to multi-user mode (text mode)
sudo systemctl isolate multi-user.target
# Verify you're in multi-user mode
systemctl list-units --type=target --state=active
# Switch back to graphical mode
sudo systemctl isolate graphical.target
# Set multi-user as permanent default (optional)
sudo systemctl set-default multi-user.target| Operation | Traditional Command | systemd Command |
|---|---|---|
| Reboot System | reboot |
systemctl reboot |
| Power Off | poweroff |
systemctl poweroff |
| Halt System | halt |
systemctl halt |
| Suspend | pm-suspend |
systemctl suspend |
| Hibernate | pm-hibernate |
systemctl hibernate |
Booting into Rescue Mode
Rescue mode provides a minimal environment with basic system services running, allowing you to repair your system when it won't boot normally. It's particularly useful for fixing configuration issues, recovering data, or resetting passwords.
Methods to Enter Rescue Mode
Method 1: From GRUB2 Boot Menu (Recommended)
- 1 Interrupt GRUB2 Boot Menu
When the GRUB2 menu appears, press any key to stop the automatic boot countdown.
- 2 Edit Boot Entry
Highlight the desired kernel entry and press 'e' to edit.
- 3 Modify Kernel Parameters
Find the line starting with
linuxorlinuxefi. Move to the end of this line and add:systemd.unit=rescue.targetAlternatively, you can use the shorter form:
rd.breakThe
rd.breakmethod breaks into the initramfs shell before the root filesystem is mounted. - 4 Boot with Modified Parameters
Press
Ctrl+XorF10to boot with the modified parameters.
Method 2: Using systemctl (From Running System)
sudo systemctl rescueThis will immediately switch to rescue mode, terminating all non-essential services and providing a root shell.
Practice Lab 2: GRUB2 Rescue Mode
Boot your system into rescue mode using GRUB2:
- Reboot your system
- When GRUB2 menu appears, press any key to stop countdown
- Press 'e' to edit the default boot entry
- Find the line starting with
linux - Add
systemd.unit=rescue.targetat the end of that line - Press
Ctrl+Xto boot - You'll be prompted for the root password (if set)
- Once in rescue mode, explore the minimal environment:
# Check mounted filesystems
mount | grep -v tmpfs
# Check available disk space
df -h
# List running processes
ps aux
# Exit rescue mode to continue normal boot
exitsingle or 1 which are aliases for rescue mode in RHEL 8.
Resetting the Root Password
Forgotten root passwords are a common administrative challenge. RHEL 8 provides secure methods to reset the root password using rescue mode or the initramfs shell.
Method 1: Using rd.break (Recommended for RHEL 8)
This method is preferred because it works even if the root account is locked or if SELinux is enforcing.
- 1 Boot with rd.break Parameter
From GRUB2 menu, edit the kernel line and add
rd.breakat the end. - 2 Access initramfs Shell
After booting, you'll get a shell prompt in the initramfs environment:
switch_root:/# - 3 Remount Root Filesystem as Read-Write
mount -o remount,rw /sysroot chroot /sysrootThe
chrootcommand changes the root directory to your actual system. - 4 Reset Root Password
passwd rootEnter the new password twice when prompted.
- 5 Handle SELinux Context (Critical!)
touch /.autorelabelThis creates a file that tells SELinux to relabel all files on next boot. Without this, your system may fail to boot properly due to incorrect security contexts.
- 6 Exit and Continue Boot
exit exitThe system will now boot normally and automatically relabel files.
Method 2: Using rescue.target
This method works if you can boot into rescue mode and know the root password for rescue mode.
- Boot with
systemd.unit=rescue.target - Enter root password when prompted
- Run
passwd rootto change the password - Type
exitto continue normal boot
⚠️ Critical Security Note
The rd.break method requires physical or console access to the system. In production environments, ensure your BIOS/UEFI and GRUB2 are password-protected to prevent unauthorized password resets.
Practice Lab 3: Root Password Recovery
Practice resetting the root password in a test environment:
# From GRUB2 menu, add 'rd.break' to kernel line
# After getting initramfs shell:
mount -o remount,rw /sysroot
chroot /sysroot
passwd root
# Enter new password: MyNewSecurePassword123!
# Confirm password: MyNewSecurePassword123!
touch /.autorelabel
exit
exit
# System will boot and relabel files (takes several minutes)
# Login with new root passwordEmergency Mode and Early Debug Shell
Emergency mode provides an even more minimal environment than rescue mode, with only the root filesystem mounted and no other services running. The early debug shell allows you to troubleshoot issues during the initramfs stage.
Emergency Mode
Emergency mode is useful when rescue mode fails to start or when you need absolute minimal system state.
- 1 Boot into Emergency Mode
From GRUB2 menu, edit kernel line and add:
systemd.unit=emergency.target - 2 Root Password Required
You'll be prompted for the root password. If you don't know it, you'll need to reset it first using the
rd.breakmethod. - 3 Minimal Environment
In emergency mode, only the root filesystem is mounted. No network, no other filesystems, no services.
Early Debug Shell
The early debug shell provides access during the initramfs stage, before the root filesystem is mounted. This is crucial for debugging storage or filesystem issues.
- 1 Enable Early Debug Shell
From GRUB2 menu, edit kernel line and add:
rd.debugThis enables verbose debugging output and provides a shell if the boot process fails.
- 2 Alternative: Force Debug Shell
To always get a debug shell during initramfs:
rd.break rd.debug - 3 Debug Shell Environment
In the debug shell, you have access to basic commands like
ls,cat,blkid,mount, etc.# List available block devices ls /dev/[sv]d* # Check filesystem types blkid # View boot logs cat /run/initramfs/rdsosreport.txt # Manually mount root filesystem mount /dev/sda2 /mnt/sysroot
Advanced Lab: Debugging Boot Failures
Simulate and debug a boot failure scenario:
# From GRUB2 menu, add 'rd.debug rd.break' to kernel line
# After getting debug shell:
# Check if root device exists
ls -l /dev/disk/by-uuid/
# Verify root= parameter matches actual UUID
cat /proc/cmdline
# Check if filesystem is corrupted
fsck -n /dev/sda2
# If fsck shows errors, repair filesystem (see next section)
# Manually mount root to verify contents
mount /dev/sda2 /mnt/sysroot
ls /mnt/sysroot/etc/fstab
# Exit debug shell to continue boot
exitRepairing File System Issues at Boot
File system corruption is one of the most common causes of boot failures. RHEL 8 provides several methods to repair filesystem issues, both automatically and manually.
Automatic File System Check (fsck)
RHEL 8 automatically runs fsck during boot under certain conditions:
- Filesystem was not cleanly unmounted (dirty bit set)
- Maximum mount count reached (default: -1, disabled)
- Maximum time between checks reached (default: 180 days)
Manual File System Repair
When automatic fsck fails or you need to force a check, manual intervention is required.
- 1 Boot into Rescue Mode
Use
rd.breakorsystemd.unit=rescue.targetto get a shell. - 2 Identify Problematic Filesystem
# List all block devices lsblk # Check filesystem types and UUIDs blkid # View /etc/fstab from your system cat /sysroot/etc/fstab - 3 Unmount the Filesystem
# If in rescue mode, filesystems may be auto-mounted umount /dev/sda2Filesystems must be unmounted before running fsck.
- 4 Run File System Check
# Check filesystem without making changes fsck -n /dev/sda2 # Repair filesystem interactively fsck -y /dev/sda2 # For XFS filesystems (common in RHEL 8) xfs_repair /dev/sda2Filesystem-Specific Tools
- ext4:
fsck.ext4,e2fsck - XFS:
xfs_repair,xfs_check - Btrfs:
btrfs check
- ext4:
- 5 Verify Repair Success
# Mount filesystem to verify mount /dev/sda2 /mnt ls /mnt umount /mnt - 6 Continue Normal Boot
exit # From rescue mode # or exit exit # From rd.break mode
Practice Lab 4: File System Repair
Practice repairing a simulated filesystem corruption:
# WARNING: Only perform on test systems!
# First, create a test filesystem
dd if=/dev/zero of=/tmp/testfs.img bs=1M count=100
mkfs.ext4 /tmp/testfs.img
# Mount and create some files
mkdir /tmp/testmount
mount -o loop /tmp/testfs.img /tmp/testmount
echo "test data" > /tmp/testmount/testfile.txt
umount /tmp/testmount
# Simulate corruption by editing image directly
hexdump -C /tmp/testfs.img | head -20
# (In real scenario, corruption would be accidental)
# Run fsck to repair
fsck -y /tmp/testfs.img
# Verify repair
mount -o loop /tmp/testfs.img /tmp/testmount
cat /tmp/testmount/testfile.txt
umount /tmp/testmountReal-World Scenario: Production Server Recovery
Your production web server won't boot due to filesystem corruption:
# Boot with rd.break
# Remount root as read-write
mount -o remount,rw /sysroot
# Check which filesystems are problematic
cat /sysroot/etc/fstab
blkid
# Unmount all filesystems except root
umount /sysroot/boot
umount /sysroot/home
# Run fsck on root filesystem
fsck -y /dev/sda2
# Run fsck on other filesystems
fsck -y /dev/sda1 # /boot
fsck -y /dev/sda3 # /home
# If using LVM, check volume groups
vgscan
vgchange -ay
# Verify all filesystems mount correctly
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
mount /dev/sda3 /mnt/home
ls /mnt/var/log
umount /mnt/{boot,home,}
# Continue boot
touch /sysroot/.autorelabel
exit
exitAdvanced Boot Troubleshooting
Common Boot Issues and Solutions
Issue: "No Root Device Found"
Cause: Kernel can't find the root filesystem specified in boot parameters.
Solution:
# Boot with rd.break
# Check actual root device UUID
blkid | grep -i root
# Compare with boot parameter
cat /proc/cmdline | grep root=
# If UUID changed, update /etc/fstab and GRUB configuration
# Or boot with correct root= parameter temporarilyIssue: "Kernel Panic - Not Syncing"
Cause: Critical kernel error, often due to missing modules or hardware issues.
Solution:
# Boot with older kernel from GRUB menu
# Or boot with rd.debug to see detailed error messages
# Check hardware (RAM test, disk health)
# Reinstall kernel packages if corruptedIssue: "Timed Out Waiting for Device"
Cause: systemd can't find a device specified in /etc/fstab.
Solution:
# Boot into rescue mode
# Check /etc/fstab for incorrect device entries
# Comment out problematic lines temporarily
# Or update device UUIDs to match current system
blkid
vi /etc/fstabGRUB2 Recovery
If GRUB2 itself is corrupted, you can reinstall it:
# From rescue mode or live CD
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot # if separate /boot
# Chroot into system
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
chroot /mnt
# Reinstall GRUB2
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg
# Exit chroot and reboot
exit
rebootBest Practices and Security Considerations
Security Best Practices
- Password-protect GRUB2 - Prevent unauthorized boot parameter changes
- Secure BIOS/UEFI - Set admin password and disable boot from external devices
- Regular backups - Maintain system backups to recover from catastrophic failures
- Document procedures - Keep recovery procedures documented and tested
- Limit physical access - Control who can access system console
Operational Best Practices
- Test recovery procedures - Regularly practice boot recovery in test environments
- Monitor system logs for early warning signs of filesystem issues
- Keep system updated with latest kernel and systemd packages
- Maintain multiple kernel versions in GRUB2 menu
- Use LVM for flexible storage management and snapshots
⚠️ Critical Reminders
- Always create
/.autorelabelafter password reset in RHEL 8 - Never run fsck on mounted filesystems
- Backup /etc/fstab before making changes
- Test GRUB2 configuration changes before relying on them
- Document your system's specific recovery procedures
Quick Reference & Troubleshooting
🔧 Essential Boot Commands
# Check current target
systemctl get-default
# Change default target
systemctl set-default multi-user.target
# Switch to rescue mode
systemctl rescue
# Reboot system
systemctl reboot
# Power off system
systemctl poweroff
# List all targets
systemctl list-units --type=target
# Check filesystem (unmounted)
fsck -y /dev/sda2
# Repair XFS filesystem
xfs_repair /dev/sda2
# View boot logs
journalctl -b
# View previous boot logs
journalctl -b -1📌 GRUB2 Boot Parameters Reference
systemd.unit=rescue.target- Boot into rescue modesystemd.unit=emergency.target- Boot into emergency moderd.break- Break into initramfs shellrd.debug- Enable debug output and shell on failuresingleor1- Alias for rescue mode3- Boot into multi-user mode5- Boot into graphical modequiet- Suppress most boot messagesrhgb- Red Hat Graphical Boot (splash screen)
📚 Additional Resources
man systemd.special- Documentation for special systemd targetsman grub- GRUB bootloader documentationman fsck- File system check documentationman xfs_repair- XFS repair documentation/usr/share/doc/systemd/- systemd documentationjournalctl -b- View current boot logs