How to Synchronize Files with Rsync

Rsync is the gold standard for efficient file synchronization and transfer in Unix environments. It uses delta encoding to transfer only changed parts of files, making it ideal for backups, deployments, and mirroring. This guide covers practical rsync usage from a senior developer's perspective.

Why Rsync

Rsync offers significant advantages:

1. Delta Transfer: Only sends file differences
2. Compression: Reduces bandwidth usage
3. Preservation: Maintains permissions, timestamps, symlinks
4. Incremental Backups: Efficient for regular syncs
5. SSH Integration: Secure transfers over a network

Basic Syntax

rsync [options] source destination

Local Synchronization

# Copy files to another directory
rsync -av /source/path/ /destination/path/
# Mirror directory (delete extra files in the destination)
rsync -av --delete /source/ /destination/

Important: A trailing slash on the source matters:

• /source/ - copies contents of source
• /source - copies the source directory itself

Common Options

Option	Description
-a	Archive mode (recursive, preserves everything)
-v	Verbose output
-z	Compress during transfer
-P	Progress + partial (resume transfers)
-n	Dry run (show what would happen)
--delete	Delete files in dest not in source
--exclude	Exclude files matching pattern
--include	Include files matching pattern
-e	Specify remote shell

Remote Synchronization

Over SSH

# Push to remote server
rsync -avz -e ssh /local/path/ user@server:/remote/path/
# Pull from remote server
rsync -avz -e ssh user@server:/remote/path/ /local/path/
# With a non-standard SSH port
rsync -avz -e 'ssh -p 2222' /local/ user@server:/remote/
# With an SSH key
rsync -avz -e 'ssh -i ~/.ssh/mykey' /local/ user@server:/remote/

With Password

For scripts (less secure than keys):

# Using sshpass
sshpass -p "password" rsync -avz /local/ user@server:/remote/
# Or with an environment variable
SSHPASS="password" sshpass -e rsync -avz /local/ user@server:/remote/

Backup Strategies

Basic Backup

#!/bin/bash
SOURCE="/home/user/documents/"
BACKUP="/backup/documents/"
LOG="/var/log/backup.log"
rsync -av --delete \
--log-file="$LOG" \
"$SOURCE" "$BACKUP"

Incremental Backup with Hard Links

Create space-efficient incremental backups:

#!/bin/bash
DATE=$(date +%Y-%m-%d)
SOURCE="/home/user/"
BACKUP_BASE="/backup"
LATEST="$BACKUP_BASE/latest"
TARGET="$BACKUP_BASE/$DATE"
# Create backup using hard links to the previous
rsync -av --delete \
--link-dest="$LATEST" \
"$SOURCE" "$TARGET"
# Update latest symlink
rm -f "$LATEST"
ln -s "$TARGET" "$LATEST"

Remote Backup Script

#!/bin/bash
set -e
# Configuration
SOURCE_SSH="user@production:/var/www"
BACKUP_BASE="/backups/production"
DATE=$(date +%Y-%m-%d_%H%M)
LATEST="$BACKUP_BASE/latest"
TARGET="$BACKUP_BASE/$DATE"
KEEP_DAYS=7
OPTIONS="-avz --delete --progress"
OPTIONS="$OPTIONS --exclude='*.log'"
OPTIONS="$OPTIONS --exclude='node_modules'"
OPTIONS="$OPTIONS --exclude='.git'"
# Create backup
mkdir -p "$TARGET"
rsync $OPTIONS \
--link-dest="$LATEST" \
-e 'ssh -o StrictHostKeyChecking=no' \
"$SOURCE_SSH" "$TARGET"
# Update latest link
rm -f "$LATEST"
ln -s "$TARGET" "$LATEST"
# Clean up old backups
find "$BACKUP_BASE" -maxdepth 1 -type d -mtime +$KEEP_DAYS -exec rm -rf {} \;
echo "Backup completed: $TARGET"

Selective Synchronization

Include/Exclude Patterns

# Exclude specific patterns
rsync -av \
--exclude='*.log' \
--exclude='*.tmp' \
--exclude='node_modules' \
--exclude='.git' \
/source/ /dest/
# Exclude from file
rsync -av --exclude-from='exclude.txt' /source/ /dest/

exclude.txt:

*.log
*.tmp
*.swp
node_modules/
.git/
.env

Include Only Specific Files

rsync -avm \
--include='*/' \
--include='*.php' \
--include='*.js' \
--include='*.css' \
--exclude='*' \
/source/ /dest/

The -m flag prunes empty directories.

Backup Only Source Code

#!/bin/bash
rsync -avm \
--include='*/' \
--include='*.php' \
--include='*.js' \
--include='*.ts' \
--include='*.vue' \
--include='*.css' \
--include='*.scss' \
--include='*.html' \
--include='*.json' \
--include='*.env.example' \
--include='.gitignore' \
--exclude='*' \
user@server:/var/www/app/ /backup/code/

Performance Tuning

Bandwidth Limiting

# Limit to 1000 KB/s
rsync -avz --bwlimit=1000 /source/ /dest/
# Limit to 5 MB/s
rsync -avz --bwlimit=5000 /source/ /dest/

Compression

# Default compression
rsync -avz /source/ user@server:/dest/
# Skip compression for pre-compressed files
rsync -avz --skip-compress=gz/jpg/mp4/zip /source/ user@server:/dest/

Parallel Transfers

Use multiple rsync processes for many small files:

#!/bin/bash
SOURCE="/source"
DEST="user@server:/dest"
# Find directories and sync in parallel
find "$SOURCE" -maxdepth 1 -type d | \
parallel -j 4 rsync -avz {} "$DEST/"

Deployment Patterns

Simple Deployment

#!/bin/bash
rsync -avz --delete \
--exclude='.git' \
--exclude='.env' \
--exclude='node_modules' \
--exclude='storage/logs/*' \
./ user@server:/var/www/app/

Deployment with Atomic Switching

#!/bin/bash
SERVER="user@production"
APP_PATH="/var/www"
RELEASE=$(date +%Y%m%d%H%M%S)
# Sync to new release directory
rsync -avz --delete \
--exclude='.git' \
--exclude='node_modules' \
./ "$SERVER:$APP_PATH/releases/$RELEASE/"
# Run remote commands
ssh "$SERVER" << EOF
cd $APP_PATH/releases/$RELEASE
# Link shared files
ln -s $APP_PATH/shared/.env .env
ln -s $APP_PATH/shared/storage storage
# Install dependencies
composer install --no-dev
npm ci && npm run build
# Switch symlink atomically
ln -sfn $APP_PATH/releases/$RELEASE $APP_PATH/current
# Restart services
sudo systemctl reload php-fpm
# Clean up old releases (keep last 5)
ls -dt $APP_PATH/releases/*/ | tail -n +6 | xargs rm -rf
EOF

Monitoring and Logging

Progress and Statistics

# Show progress
rsync -avP /source/ /dest/
# Show statistics at end
rsync -av --stats /source/ /dest/
# Itemize changes (detailed)
rsync -avvi /source/ /dest/

Logging to File

rsync -av \
--log-file=/var/log/rsync.log \
--log-file-format="%t %f %b" \
/source/ /dest/

Troubleshooting

Dry Run First

Always test with -n:

rsync -avn --delete /source/ /dest/

Debug Connection Issues

# Verbose SSH
rsync -avz -e 'ssh -v' /source/ user@server:/dest/
# Check what would be transferred
rsync -avni /source/ /dest/

Handle Large Files

# Resume partial transfers
rsync -avP --partial /source/ /dest/
# Checksum instead of mod-time (slower but accurate)
rsync -avc /source/ /dest/

Permission Issues

# Preserve permissions (default with -a)
rsync -av /source/ /dest/
# Don't preserve (copy as current user)
rsync -rv --no-perms --no-owner --no-group /source/ /dest/

Integration with Cron

Automated Backup

# /etc/cron.d/backup
0 2 * * * root /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Lock to Prevent Overlap

#!/bin/bash
LOCKFILE="/var/run/backup.lock"
(
flock -n 200 || { echo "Backup already running"; exit 1; }
rsync -av /source/ /dest/
) 200>"$LOCKFILE"

Key Takeaways

1. Always test with -n: Dry run before real transfers
2. Trailing slash matters: /source/ vs /source
3. Use -P for large transfers: Progress and resume capability
4. --delete with caution: Can remove files in destination
5. --link-dest for backups: Space-efficient incremental backups
6. Bandwidth limiting: Be considerate on shared networks

Rsync is indispensable for any operations work—master it for efficient, reliable file synchronization across your infrastructure.