Introduction
Systemd is the standard service manager on modern Linux distributions. Understanding systemd is essential for deploying and managing applications in production—from starting services at boot to monitoring their health and managing dependencies. This guide covers practical systemd administration for application deployment.
Service Management Basics
Common Commands
# Start, stop, or restart the service
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx
# Reload configuration without restarting
sudo systemctl reload nginx
# Enable or disable the service at boot
sudo systemctl enable nginx
sudo systemctl disable nginx
# Enable and start in one command
sudo systemctl enable --now nginx
# Check service status
sudo systemctl status nginx
# List all services
sudo systemctl list-units --type=service
# List enabled services
sudo systemctl list-unit-files --type=service --state=enabled
Service Status Output
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-01-15 10:30:00 UTC; 5h ago
Docs: man:nginx(8)
Main PID: 1234 (nginx)
Tasks: 5 (limit: 4915)
Memory: 12.5M
CPU: 1.234s
CGroup: /system.slice/nginx.service
├─1234 nginx: master process /usr/sbin/nginx
└─1235 nginx: worker process
Key indicators:
- Active: Current state (running, dead, failed)
- Main PID: Process ID of the main process
- Memory/CPU: Resource usage
Creating Custom Services
Basic Service Unit
# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
Documentation=https://myapp.example.com/docs
After=network.target
[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node /var/www/myapp/server.js
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
# Apply changes
sudo systemctl daemon-reload
sudo systemctl enable --now myapp
Service Types
# Simple (default): The main process is the service
Type=simple
ExecStart=/usr/bin/myapp
# Forking: The process forks and the parent exits
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/bin/myapp --daemon
# Oneshot: The process runs once and exits
Type=oneshot
ExecStart=/usr/bin/setup-script.sh
RemainAfterExit=yes
# Notify: The process signals ready
Type=notify
ExecStart=/usr/bin/myapp --notify
Environment Configuration
[Service]
# Inline environment variables
Environment=NODE_ENV=production
Environment=PORT=3000
# Environment file
EnvironmentFile=/etc/myapp/environment
# Multiple environment files
EnvironmentFile=-/etc/myapp/environment # - means optional
EnvironmentFile=/etc/myapp/secrets
# /etc/myapp/environment
NODE_ENV=production
PORT=3000
DATABASE_URL=postgres://localhost/myapp
Resource Limits
Memory and CPU
[Service]
# Memory limits
MemoryMax=512M
MemoryHigh=400M
# CPU limits
CPUQuota=50%
CPUWeight=100
# File descriptors
LimitNOFILE=65535
# Process limits
LimitNPROC=4096
# Core dumps
LimitCORE=infinity
I/O Limits
[Service]
IOWeight=500
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M
Security Hardening
Isolation Options
[Service]
# Run as non-root
User=myapp
Group=myapp
# Filesystem restrictions
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp
# Temporary directory
PrivateTmp=yes
# Network isolation
PrivateNetwork=yes # No network access
# Device access
PrivateDevices=yes
# Prevent privilege escalation
NoNewPrivileges=yes
# System call filtering
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
Capability Restrictions
[Service]
# Drop all capabilities except the needed ones
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
# Remove all capabilities
CapabilityBoundingSet=
Process Supervision
Restart Policies
[Service]
# Restart options: no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, always
Restart=on-failure
# Time between restarts
RestartSec=5s
# Maximum restart attempts
StartLimitIntervalSec=300
StartLimitBurst=5
# Actions when the limit is reached
# FailureAction=reboot
# SuccessAction=poweroff
Watchdog
[Service]
Type=notify
WatchdogSec=30s
# The application must call sd_notify(0, "WATCHDOG=1") periodically
Pre/Post Commands
[Service]
ExecStartPre=/usr/bin/check-config.sh
ExecStart=/usr/bin/myapp
ExecStartPost=/usr/bin/notify-started.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/bin/graceful-stop.sh
ExecStopPost=/usr/bin/cleanup.sh
Logging with Journal
Viewing Logs
# Service logs
sudo journalctl -u myapp
# Follow logs
sudo journalctl -u myapp -f
# Last 100 lines
sudo journalctl -u myapp -n 100
# Since boot
sudo journalctl -u myapp -b
# Time range
sudo journalctl -u myapp --since "2024-01-15 10:00:00" --until "2024-01-15 12:00:00"
# Priority filter (err and above)
sudo journalctl -u myapp -p err
# JSON output
sudo journalctl -u myapp -o json
# Disk usage
sudo journalctl --disk-usage
Log Configuration
[Service]
# Send output to the journal
StandardOutput=journal
StandardError=journal
# Or to a file
StandardOutput=append:/var/log/myapp/output.log
StandardError=append:/var/log/myapp/error.log
# Syslog identifier
SyslogIdentifier=myapp
Journal Persistence
# Make logs persistent
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
# Configure retention
sudo vi /etc/systemd/journald.conf
# SystemMaxUse=500M
# MaxRetentionSec=1month
sudo systemctl restart systemd-journald
Timers (Cron Replacement)
Timer Unit
# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/backup.service
[Unit]
Description=Backup service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
Timer Schedules
# Calendar expressions
OnCalendar=hourly
OnCalendar=daily
OnCalendar=weekly
OnCalendar=monthly
OnCalendar=*-*-* 00:00:00 # Daily at midnight
OnCalendar=Mon *-*-* 00:00:00 # Every Monday
OnCalendar=*-*-* *:00:00 # Every hour
OnCalendar=*-*-* *:*:00 # Every minute
# Relative timing
OnBootSec=5min # 5 minutes after boot
OnUnitActiveSec=1h # 1 hour after last activation
Managing Timers
# List timers
sudo systemctl list-timers
# Enable timer
sudo systemctl enable --now backup.timer
# Run the associated service immediately
sudo systemctl start backup.service
Socket Activation
# /etc/systemd/system/myapp.socket
[Unit]
Description=My App Socket
[Socket]
ListenStream=8080
Accept=no
[Install]
WantedBy=sockets.target
# /etc/systemd/system/myapp.service
[Unit]
Description=My App
Requires=myapp.socket
[Service]
ExecStart=/usr/bin/myapp
StandardInput=socket
The service only starts when a connection is received on the socket.
Multi-Instance Services
# /etc/systemd/system/[email protected]
[Unit]
Description=My App instance %i
[Service]
ExecStart=/usr/bin/myapp --instance %i --port %i
User=www-data
[Install]
WantedBy=multi-user.target
# Start multiple instances
sudo systemctl enable --now myapp@3000
sudo systemctl enable --now myapp@3001
sudo systemctl enable --now myapp@3002
# Manage all instances
sudo systemctl restart 'myapp@*'
Debugging
# Verify unit file syntax
sudo systemd-analyze verify /etc/systemd/system/myapp.service
# Show unit configuration
sudo systemctl show myapp
# Show unit dependencies
sudo systemctl list-dependencies myapp
# Check boot performance
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain myapp.service
Best Practices
- Use
Type=notify when possible for accurate status - Set appropriate restart policies for resilience
- Apply security hardening options
- Use environment files for configuration
- Configure resource limits to prevent runaway processes
- Use timers instead of cron for scheduled tasks
- Monitor with journalctl and set log retention
Conclusion
Systemd provides powerful service management capabilities. Create proper unit files with security hardening, configure appropriate restart policies, and use timers for scheduled tasks. The patterns in this guide help you deploy and manage reliable production services.
