Cyber Services / All lessons / CentOS / Linux Administration / Advanced Storage Features in CentOS 8.5

Advanced Storage Features in CentOS 8.5

Module 19 of 33 · 16 min read · Free to read

MODULES ↓ Command blocks have a COPY button

Master Stratis Storage Management and VDO (Virtual Data Optimizer) for Thin Provisioning, Snapshots, and Data Deduplication

Modern storage requirements demand more than basic partitioning and filesystem management. CentOS 8.5 introduces two powerful advanced storage technologies: Stratis for simplified storage pool management with built-in snapshots, and VDO (Virtual Data Optimizer) for transparent data deduplication, compression, and thin provisioning.

This comprehensive guide covers installation, configuration, management, and real-world scenarios for both Stratis and VDO. You'll learn how to create thin-provisioned filesystems that grow dynamically, take instantaneous snapshots, implement storage-level data deduplication, and optimize storage utilization. All procedures are tested and verified on CentOS 8.5.

Understanding Advanced Storage Concepts

The Storage Management Analogy

Think of traditional storage like managing individual filing cabinets — each has fixed space and you need to plan carefully. Stratis is like a smart storage room where you can instantly create new filing cabinets of any size from a shared pool, take snapshots of entire cabinets, and they automatically expand as needed. VDO is like a magical compression system that automatically removes duplicate documents and compresses everything, giving you much more usable space than your physical storage suggests.

Why Advanced Storage Features Matter

  • Storage Efficiency - Thin provisioning and deduplication maximize storage utilization
  • Operational Simplicity - Automated management reduces administrative overhead
  • Snapshot Capabilities - Instantaneous point-in-time copies for backup and recovery
  • Dynamic Growth - Filesystems automatically expand as data is written
  • Cost Savings - Reduce physical storage requirements through optimization
FeatureTraditional LVMStratisVDO
Thin Provisioning Manual setup required Built-in and automatic Built-in and automatic
Snapshots Manual, space-intensive Automatic, space-efficient Not available
Data Deduplication Not available Not available Built-in block-level
Compression Not available Not available Built-in (LZ4)
Management Complexity High (multiple commands) Low (single tool) Medium (dedicated tool)
Use Case Basic volume management Application storage with snapshots Backup targets, VM storage

Stratis Storage Management

Stratis is a next-generation storage management solution that simplifies the creation and management of storage pools and filesystems. It provides automatic thin provisioning, snapshots, and monitoring capabilities with a simple command-line interface.

Stratis Architecture

Pool - Collection of block devices (physical storage)

Filesystem - Thin-provisioned filesystem created from pool space

Blockdev - Individual physical devices added to pools

Snapshot - Point-in-time copy of a filesystem

Installing and Configuring Stratis

  1. 1 Install Stratis Packages
    sudo dnf install -y stratis-cli stratisd

    This installs both the daemon (stratisd) and command-line tool (stratis).

  2. 2 Enable and Start Stratis Daemon
    sudo systemctl enable --now stratisd
    sudo systemctl status stratisd

    Verify the daemon is active and running.

  3. 3 Verify Installation
    stratis --version
    stratis pool list

    The second command should return empty if no pools exist yet.

Practice Lab 1: Stratis Installation

Install and verify Stratis on your CentOS 8.5 system:

# Install Stratis packages
sudo dnf install -y stratis-cli stratisd

# Start and enable the daemon
sudo systemctl enable --now stratisd

# Verify the service is running
sudo systemctl status stratisd --no-pager

# Check Stratis version and list pools
stratis --version
stratis pool list

# Expected output for empty system:
# Pool Name   Total Physical Size   Total Physical Used
# no pools present in the pool namespace

Managing Stratis File Systems

Once Stratis is installed, you can create storage pools and filesystems to manage your data efficiently.

Creating and Managing Stratis Pools

  1. 1 Identify Available Block Devices
    lsblk
    sudo fdisk -l

    Identify unused block devices (e.g., /dev/sdb, /dev/sdc). Ensure they don't contain important data as Stratis will format them.

  2. 2 Create a Stratis Pool
    sudo stratis pool create mypool /dev/sdb

    You can add multiple devices:

    sudo stratis pool create mypool /dev/sdb /dev/sdc
  3. 3 Verify Pool Creation
    stratis pool list
    stratis blockdev list
  4. 4 Create a Filesystem
    sudo stratis filesystem create mypool myfs

    Stratis automatically creates a thin-provisioned XFS filesystem.

  5. 5 List Filesystems
    stratis filesystem list

    This shows all Stratis filesystems with their pool, name, used size, and UUID.

Practice Lab 2: Stratis Filesystem Management

Create a Stratis pool and filesystem:

# Identify available disks (use virtual disks in lab)
lsblk

# Create pool with one or more disks
sudo stratis pool create datapool /dev/sdb

# Verify pool creation
stratis pool list
# Output should show:
# Pool Name   Total Physical Size   Total Physical Used
# datapool    10.00 GiB             54.00 MiB

# Create filesystem
sudo stratis filesystem create datapool webdata

# List filesystems
stratis filesystem list
# Output should show:
# Pool Name   Name      Used      Created            Device
# datapool    webdata   546 MiB   Nov 15 2023 10:30  /stratis/datapool/webdata

Filesystem Properties and Monitoring

Stratis filesystems are thin-provisioned, meaning they can be larger than the available physical space. The filesystem automatically grows as data is written, up to the pool's capacity.

Understanding Stratis Space Usage

  • Total Physical Size - Total size of all block devices in the pool
  • Total Physical Used - Actual space used on physical devices
  • Filesystem Used - Logical space used by the filesystem
  • Thin Provisioning - Filesystem size can exceed physical space

Persistently Mounting Stratis File Systems

Stratis filesystems need to be mounted persistently to survive reboots. Unlike traditional filesystems, Stratis uses a special mount mechanism.

Mounting Stratis Filesystems

  1. 1 Create Mount Point
    sudo mkdir -p /mnt/webdata
  2. 2 Mount Using Filesystem Path
    sudo mount /stratis/datapool/webdata /mnt/webdata

    Stratis filesystems are accessible via /stratis/<pool>/<filesystem>.

  3. 3 Get Filesystem UUID
    stratis filesystem list

    Note the UUID for the filesystem you want to mount persistently.

  4. 4 Add to /etc/fstab
    echo "UUID=<filesystem-uuid> /mnt/webdata xfs defaults,x-systemd.requires=stratisd.service 0 0" | sudo tee -a /etc/fstab

    The x-systemd.requires=stratisd.service option ensures Stratis daemon starts before mounting.

  5. 5 Test Mount Configuration
    sudo umount /mnt/webdata
    sudo mount -a
    df -h /mnt/webdata

Practice Lab 3: Persistent Mounting

Configure persistent mounting for your Stratis filesystem:

# Create mount point
sudo mkdir -p /var/www/html

# Get filesystem UUID
stratis filesystem list
# Example output UUID: 12345678-1234-1234-1234-123456789012

# Add to fstab (replace UUID with actual value)
echo "UUID=12345678-1234-1234-1234-123456789012 /var/www/html xfs defaults,x-systemd.requires=stratisd.service 0 0" | sudo tee -a /etc/fstab

# Test the configuration
sudo umount /stratis/datapool/webdata
sudo mount -a

# Verify mount
df -h /var/www/html
ls -la /var/www/html
Important: Always use the filesystem UUID in /etc/fstab rather than the /stratis path, as the path may not be available during early boot before stratisd starts.

Stratis Filesystem Snapshots

Snapshots are one of Stratis's most powerful features, allowing you to create instantaneous point-in-time copies of filesystems with minimal space overhead.

Creating and Managing Snapshots

  1. 1 Create a Snapshot
    sudo stratis filesystem snapshot datapool webdata webdata-snap-20231115

    Syntax: stratis filesystem snapshot <pool> <original-fs> <snapshot-name>

  2. 2 Verify Snapshot Creation
    stratis filesystem list

    The snapshot will appear as a separate filesystem with the same pool.

  3. 3 Mount the Snapshot
    sudo mkdir -p /mnt/snapshot
    sudo mount /stratis/datapool/webdata-snap-20231115 /mnt/snapshot
  4. 4 Access Snapshot Data
    ls -la /mnt/snapshot
    cat /mnt/snapshot/somefile.txt

    Snapshots are read-write by default, but changes don't affect the original filesystem.

  5. 5 Remove a Snapshot
    sudo umount /mnt/snapshot
    sudo stratis filesystem destroy datapool webdata-snap-20231115

Practice Lab 4: Snapshot Management

Create and use a Stratis snapshot:

# Create some test data in original filesystem
echo "Original data created on $(date)" | sudo tee /var/www/html/testfile.txt

# Create snapshot
sudo stratis filesystem snapshot datapool webdata webdata-backup

# Mount snapshot
sudo mkdir -p /tmp/snapshot-restore
sudo mount /stratis/datapool/webdata-backup /tmp/snapshot-restore

# Verify snapshot contains original data
cat /tmp/snapshot-restore/testfile.txt

# Modify original filesystem (snapshot remains unchanged)
echo "Modified data on $(date)" | sudo tee /var/www/html/testfile.txt

# Verify snapshot still has original data
cat /tmp/snapshot-restore/testfile.txt

# Clean up
sudo umount /tmp/snapshot-restore
sudo stratis filesystem destroy datapool webdata-backup

Real-World Scenario: Application Update Rollback

Use snapshots to safely update a web application with rollback capability:

# Before update, create snapshot
sudo stratis filesystem snapshot webpool production-app pre-update-snapshot

# Perform application update
sudo rsync -av /tmp/new-app/ /var/www/html/

# If update fails, rollback using snapshot
sudo systemctl stop httpd
sudo umount /var/www/html
sudo stratis filesystem destroy webpool production-app
sudo stratis filesystem snapshot webpool pre-update-snapshot production-app
# Remount and restart
sudo mount UUID=<new-fs-uuid> /var/www/html
sudo systemctl start httpd

# If update succeeds, remove snapshot after verification period
sudo stratis filesystem destroy webpool pre-update-snapshot

Guided Exercises: Thin Provisioning and Dynamic Growth

These exercises demonstrate the core benefits of Stratis: thin provisioning and automatic filesystem growth.

Guided Exercise 1: Create Thin-Provisioned Filesystem

Create a filesystem that appears larger than available physical space:

# Create pool with 10GB physical disk
sudo stratis pool create thinpool /dev/sdb

# Create filesystem (appears as 1TB even with 10GB pool)
sudo stratis filesystem create thinpool largefs

# Check filesystem size
df -h /stratis/thinpool/largefs
# Output shows filesystem size much larger than pool

# Verify with Stratis command
stratis filesystem list
# Shows logical size vs physical usage

Guided Exercise 2: Verify Dynamic Growth

Demonstrate automatic filesystem expansion as data is written:

# Mount the filesystem
sudo mkdir -p /mnt/largefs
sudo mount /stratis/thinpool/largefs /mnt/largefs

# Check initial usage
df -h /mnt/largefs
stratis pool list

# Create large file to consume space
sudo dd if=/dev/zero of=/mnt/largefs/bigfile bs=1M count=2000

# Check usage after writing data
df -h /mnt/largefs
stratis pool list
# Physical usage should increase while filesystem size remains constant

# Continue until pool is nearly full
sudo dd if=/dev/zero of=/mnt/largefs/bigfile2 bs=1M count=8000

# Attempt to write beyond pool capacity (should fail gracefully)
sudo dd if=/dev/zero of=/mnt/largefs/toobig bs=1M count=1000
# Write will fail when pool is exhausted

Guided Exercise 3: Create and Access Snapshot Data

Work with snapshot data in a practical scenario:

# Create database backup scenario
echo "CREATE DATABASE production;" | sudo tee /mnt/largefs/init.sql
echo "INSERT INTO users VALUES (1, 'admin');" | sudo tee -a /mnt/largefs/init.sql

# Create snapshot before major operation
sudo stratis filesystem snapshot thinpool largefs pre-migration-snap

# Simulate database migration
echo "ALTER TABLE users ADD COLUMN email VARCHAR(255);" | sudo tee -a /mnt/largefs/init.sql

# Access snapshot to verify original state
sudo mkdir -p /tmp/original-db
sudo mount /stratis/thinpool/pre-migration-snap /tmp/original-db
cat /tmp/original-db/init.sql
# Shows original data without email column

# Compare with current state
cat /mnt/largefs/init.sql
# Shows modified data with email column

# Clean up
sudo umount /tmp/original-db
sudo stratis filesystem destroy thinpool pre-migration-snap

VDO - Virtual Data Optimizer

VDO (Virtual Data Optimizer) is a Linux kernel module that provides inline data deduplication, compression, and thin provisioning. It's ideal for workloads with repetitive data like virtual machines, backups, and container images.

VDO Architecture

Physical Storage - Actual block devices (HDD/SSD)

VDO Volume - Logical volume with deduplication/compression

Logical Size - Apparent size presented to filesystem (can be larger than physical)

Index Memory - RAM used for deduplication metadata (1GB per 1TB logical)

VDO Benefits

  • Block-level Deduplication - Eliminates duplicate 4KB blocks across entire volume
  • LZ4 Compression - Compresses unique data blocks
  • Thin Provisioning - Logical size can exceed physical capacity
  • Transparent Operation - Applications see normal block device
  • Performance Optimized - Designed for minimal CPU overhead
Use Cases: VDO is ideal for backup repositories, VM storage, container registries, log archives, and any workload with repetitive data patterns. Avoid using VDO for already-compressed data (videos, images) or databases with built-in compression.

Installing and Configuring VDO

VDO requires specific packages and careful configuration to achieve optimal results.

Installation Process

  1. 1 Install VDO Packages
    sudo dnf install -y vdo kmod-kvdo

    This installs the VDO management tools and kernel module.

  2. 2 Load VDO Kernel Module
    sudo modprobe kvdo
    lsmod | grep kvdo

    Verify the module is loaded successfully.

  3. 3 Enable VDO Service
    sudo systemctl enable --now vdo
    sudo systemctl status vdo

    The VDO service manages VDO volumes at boot time.

Practice Lab 5: VDO Installation

Install and verify VDO on your CentOS 8.5 system:

# Install VDO packages
sudo dnf install -y vdo kmod-kvdo

# Load kernel module
sudo modprobe kvdo

# Verify module loaded
lsmod | grep kvdo
# Expected output: kvdo 2345678 0

# Enable and start VDO service
sudo systemctl enable --now vdo

# Verify service status
sudo systemctl status vdo --no-pager

# Check VDO version
sudo vdo --version

Creating and Formatting VDO Volumes

Creating a VDO volume requires careful consideration of physical size, logical size, and index memory requirements.

VDO Volume Creation

  1. 1 Identify Physical Device
    lsblk
    sudo fdisk -l

    Select an unused block device (e.g., /dev/sdc).

  2. 2 Calculate Logical Size

    For deduplication workloads, logical size can be 2-10x physical size:

    # Example: 10GB physical disk, 50GB logical size
    # Index memory needed: ~50MB (1GB per 1TB logical)
  3. 3 Create VDO Volume
    sudo vdo create --name=vdo1 --device=/dev/sdc --vdoLogicalSize=50G

    Key parameters:

    • --name - VDO volume name
    • --device - Physical block device
    • --vdoLogicalSize - Logical size (optional, defaults to physical size)
    • --indexMem - Index memory in GB (optional, auto-calculated)
  4. 4 Verify VDO Creation
    sudo vdo status --name=vdo1
    ls /dev/mapper/vdo1
  5. 5 Format VDO Volume
    sudo mkfs.xfs -K /dev/mapper/vdo1

    The -K flag disables discard operations which can interfere with VDO.

  6. 6 Mount VDO Volume
    sudo mkdir -p /mnt/vdo1
    sudo mount /dev/mapper/vdo1 /mnt/vdo1

Practice Lab 6: VDO Volume Creation

Create and format a VDO volume:

# Create VDO volume (10GB physical, 30GB logical)
sudo vdo create --name=backup-vdo --device=/dev/sdc --vdoLogicalSize=30G

# Verify creation
sudo vdo status --name=backup-vdo
# Should show physical size: 10.0G, logical size: 30.0G

# Check device mapper
ls -l /dev/mapper/backup-vdo

# Format with XFS (disable discard)
sudo mkfs.xfs -K /dev/mapper/backup-vdo

# Create mount point and mount
sudo mkdir -p /backup
sudo mount /dev/mapper/backup-vdo /backup

# Verify mount
df -h /backup
# Should show 30G size

Testing VDO Deduplication

Deduplication is VDO's most valuable feature. This section demonstrates how to test and measure deduplication effectiveness.

Deduplication Testing Methodology

  1. 1 Create Test Data with Duplicates
    # Create base file
    sudo dd if=/dev/urandom of=/tmp/basefile bs=1M count=100
    
    # Create multiple copies (simulates VM templates, backups, etc.)
    for i in {1..10}; do
        sudo cp /tmp/basefile /backup/copy$i
    done
  2. 2 Monitor VDO Statistics
    sudo vdostats --human-readable

    Key metrics to watch:

    • Used% - Physical space utilization
    • Logical Used - Apparent data size
    • Physical Used - Actual space consumed
  3. 3 Calculate Deduplication Ratio
    # Before adding duplicates
    sudo vdostats --human-readable
    
    # After adding duplicates  
    sudo vdostats --human-readable
    
    # Deduplication ratio = Logical Used / Physical Used
  4. 4 Test with Real Workloads
    # Copy existing data with natural duplicates
    sudo cp -r /var/log /backup/logs-original
    sudo cp -r /var/log /backup/logs-duplicate
    
    # Monitor space savings
    sudo vdostats --human-readable

Practice Lab 7: Deduplication Testing

Test VDO deduplication with controlled duplicate data:

# Check initial VDO stats
sudo vdostats --human-readable
# Example output:
# Device           Size      Used Available Use% Space saving%
# /dev/mapper/backup-vdo  30.0G  104.0M   29.9G   0%          0%

# Create base data (100MB)
sudo dd if=/dev/urandom of=/tmp/testdata bs=1M count=100

# Copy data 5 times to VDO volume
for i in {1..5}; do
    echo "Copying iteration $i"
    sudo cp /tmp/testdata /backup/duplicate$i
done

# Check VDO stats after duplication
sudo vdostats --human-readable
# Should show:
# Logical Used: ~500MB
# Physical Used: ~100MB  
# Space saving: ~80%

# Calculate actual ratio
# Space saving % = (1 - Physical/Logical) * 100

Understanding VDO Statistics

Space saving % = How much physical space is saved due to deduplication and compression

Logical Used = Total size of all data written to the VDO volume

Physical Used = Actual space consumed on the physical device

Index Memory = RAM used for deduplication metadata (critical for performance)

Extending and Removing VDO Volumes

VDO volumes can be extended to use additional physical space, and removed when no longer needed.

Extending VDO Volumes

VDO volumes can only be extended by adding more physical devices or extending the underlying device.

  1. 1 Extend Underlying Physical Device

    If using LVM or virtual disks, extend the physical device first:

    # For LVM: extend logical volume
    sudo lvextend -L+10G /dev/vg0/vdo-lv
    
    # For virtual machines: extend virtual disk, then rescan
    echo 1 | sudo tee /sys/class/block/sdc/device/rescan
  2. 2 Grow VDO Physical Size
    sudo vdo growPhysical --name=vdo1

    This tells VDO to use the additional physical space.

  3. 3 Grow VDO Logical Size (Optional)
    sudo vdo growLogical --name=vdo1 --vdoLogicalSize=60G

    Increase logical size if needed (requires available physical space).

  4. 4 Grow Filesystem
    # For XFS filesystems
    sudo xfs_growfs /backup
    
    # Verify new size
    df -h /backup

Removing VDO Volumes

  1. 1 Unmount Filesystem
    sudo umount /backup
  2. 2 Remove from /etc/fstab
    sudo sed -i '/backup/d' /etc/fstab
  3. 3 Stop VDO Volume
    sudo vdo stop --name=vdo1
  4. 4 Remove VDO Volume
    sudo vdo remove --name=vdo1

    This permanently deletes all data on the VDO volume.

Practice Lab 8: VDO Volume Management

Extend and remove a VDO volume:

# Extend physical device (assuming virtual disk was extended)
echo 1 | sudo tee /sys/class/block/sdc/device/rescan

# Grow VDO physical size
sudo vdo growPhysical --name=backup-vdo

# Optionally grow logical size
sudo vdo growLogical --name=backup-vdo --vdoLogicalSize=50G

# Grow filesystem
sudo xfs_growfs /backup

# Verify new sizes
df -h /backup
sudo vdostats --human-readable

# To remove volume:
sudo umount /backup
sudo sed -i '/backup-vdo/d' /etc/fstab
sudo vdo stop --name=backup-vdo
sudo vdo remove --name=backup-vdo

Advanced Configuration and Best Practices

Stratis Best Practices

  • Use dedicated disks - Don't mix Stratis pools with other data on same disks
  • Monitor pool usage - Set up alerts when pool usage exceeds 80%
  • Regular snapshots - Create snapshots before major system changes
  • Backup critical data - Snapshots are not backups; use separate backup strategy
  • Plan for expansion - Add disks to pools before they become full

VDO Best Practices

  • Sufficient RAM - Allocate 1GB RAM per 1TB logical size for index
  • Avoid double compression - Don't use VDO with already-compressed data
  • Monitor performance - Watch CPU usage during heavy write operations
  • Test deduplication ratio - Verify your workload benefits from deduplication
  • Use appropriate filesystem - XFS with -K flag is recommended

Performance Considerations

Performance Guidelines

Stratis: Minimal performance overhead, suitable for most workloads including databases

VDO: Write operations have CPU overhead due to deduplication/compression; read operations have minimal overhead

Memory Requirements: VDO needs significant RAM for index; Stratis has minimal memory requirements

I/O Patterns: Both work best with sequential I/O; random I/O may reduce deduplication effectiveness

⚠️ Critical Considerations

  • Never use VDO for root filesystem - Boot complications and recovery difficulties
  • Stratis snapshots are not backups - They reside on the same pool as original data
  • VDO requires careful capacity planning - Running out of physical space causes write failures
  • Both technologies are relatively new - Test thoroughly in non-production environments first
  • Backup your data - Advanced storage features don't replace proper backup strategies

Troubleshooting and Reference

🔧 Common Issues and Solutions

  • Stratis pool won't create? → Ensure devices are unused and not mounted; check dmesg for errors
  • VDO creation fails with memory error? → Insufficient RAM for index; reduce logical size or add more RAM
  • Filesystem won't mount after reboot? → Verify /etc/fstab entries and systemd dependencies
  • Deduplication not working? → Test with truly duplicate data; check VDO statistics with vdostats
  • Pool full but filesystem shows space? → Thin provisioning limit reached; add more physical devices to pool

📌 Comprehensive Command Reference

# STRATIS COMMANDS
stratis pool list                    # List all pools
stratis pool create <name> <device> # Create pool
stratis filesystem list             # List filesystems
stratis filesystem create <pool> <fs> # Create filesystem
stratis filesystem snapshot <pool> <orig> <snap> # Create snapshot
stratis filesystem destroy <pool> <fs> # Remove filesystem/snapshot

# VDO COMMANDS  
sudo vdo create --name=<name> --device=<dev> # Create VDO volume
sudo vdo status --name=<name>       # Show VDO status
sudo vdostats --human-readable      # Show space usage
sudo vdo growPhysical --name=<name> # Extend physical size
sudo vdo growLogical --name=<name> --vdoLogicalSize=<size> # Extend logical
sudo vdo stop --name=<name>         # Stop VDO volume
sudo vdo remove --name=<name>       # Remove VDO volume

# MONITORING
df -h                               # Filesystem usage
lsblk                              # Block device overview
free -h                            # Memory usage (important for VDO)
iostat -x 1                        # I/O performance monitoring

📚 Additional Resources

  • man stratis - Stratis command documentation
  • man vdo - VDO command documentation
  • man vdostats - VDO statistics documentation
  • Red Hat Storage Administration Guide - Official documentation
  • /var/log/stratisd.log - Stratis daemon logs
  • journalctl -u vdo - VDO service logs
Modules (33) ← CentOS / Linux Administration