Puppet Configuration Management: Enterprise Infrastructure Automation Guide

10 servers = Manual procedures (shell scripts, documentation)
100 servers = Some automation, inconsistent configuration
500 servers = Inconsistent nightmares (each server slightly different)
2,000+ servers = MUST have configuration management or fail

#!/bin/bash
# Step 1: Check if package installed
if ! dpkg -l | grep -q nginx; then
  # Step 2: Install package
  apt-get install -y nginx
fi
# Step 3: Start service
systemctl start nginx
# Step 4: Enable on boot
systemctl enable nginx
# Step 5: Check if config file exists
if [ ! -f /etc/nginx/nginx.conf ]; then
  # Step 6: Deploy config
  cp nginx.conf /etc/nginx/
fi

package { 'nginx':
  ensure => installed,
}

service { 'nginx':
  ensure => running,
  enable => true,
}

file { '/etc/nginx/nginx.conf':
  ensure  => file,
  content => template('nginx/nginx.conf.erb'),
}

Script (imperative): "Do step 1, then step 2, then step 3"
Problem: If step 2 fails, step 3 doesn't run
Problem: Running script twice might cause issues
Problem: Error handling is manual

Puppet (declarative): "System SHOULD have nginx running"
Benefit: Puppet handles ordering automatically
Benefit: Safe to run repeatedly (idempotent)
Benefit: Puppet handles errors intelligently

┌─────────────────────────────────────────────────────┐
│            Puppet Master (Central)                  │
│                                                     │
│  Stores:                                            │
│  ├─ Manifests (configuration definitions)           │
│  ├─ Modules (reusable components)                   │
│  ├─ Facts (system information)                      │
│  └─ Reports (what changed on each agent)            │
└─────────────────────────────────────────────────────┘
            ↑            ↑            ↑
      (pull config) (apply config) (report results)
            │            │            │
    ┌───────┴────┬───────┴────┬───────┴────┐
    │            │            │            │
  Server A    Server B    Server C    Server N
  (Agent)     (Agent)     (Agent)     (Agent)
    │            │            │            │
    ├─ nginx    ├─ nginx    ├─ nginx    ├─ nginx
    ├─ MySQL    ├─ MySQL    ├─ MySQL    ├─ MySQL
    ├─ Config   ├─ Config   ├─ Config   ├─ Config
    └─ Uniform  └─ Uniform  └─ Uniform  └─ Uniform

1. Every 30 minutes (configurable):
   Puppet Agent on Server A → Contacts Master
   Agent: "Hello Master, I'm Server A"

2. Master looks up Server A in:
   ├─ Inventory
   ├─ Node definitions
   └─ Rules

3. Master compiles Puppet manifest into "catalog"
   Catalog = "Here's what Server A should look like"

4. Master sends catalog to Agent
   Agent: "Got it"

5. Agent APPLIES catalog
   ├─ Installs packages if missing
   ├─ Updates configuration files if changed
   ├─ Starts/stops services if needed
   ├─ Changes permissions if needed
   └─ etc.

6. Agent REPORTS back to Master
   Master: "Server A is now in desired state"
   (Or: "Server A failed to reach desired state - here's why")

7. Master logs report
   You can query: "Show me all servers not in desired state"

Datacenters:
├─ US East: 150 servers
├─ US West: 150 servers
├─ Europe: 100 servers
└─ Asia: 100 servers

Requirements:
├─ All must run nginx (same version)
├─ All must have security patches
├─ All must have monitoring agent
├─ All must have log rotation
├─ All must have firewall rules (uniform)
├─ All must have specific kernel parameters
└─ All must report to central monitoring

Manual approach:
1. Write deployment script
2. SSH to each server
3. Run script
4. Hope it works on all variants
5. Debug 10 servers that failed
6. Update servers as requirements change
7. Every server slightly different (configs drift over time)

Result: Chaos. 3am emergency calls.

Define once (in master):
├─ nginx package + version
├─ nginx configuration
├─ monitoring agent
├─ security patches
├─ kernel parameters
└─ firewall rules

Deploy:
├─ Write once
├─ Push to 500 servers automatically
├─ Puppet agent applies on each server
├─ Reports back results
├─ You monitor compliance dashboard

Result: 500 identical servers, 0 manual SSH logins.

# A package resource
package { 'nginx':
  ensure => installed,
}

# A service resource
service { 'nginx':
  ensure => running,
  enable => true,
}

# A file resource
file { '/etc/nginx/nginx.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0644',
  content => 'nginx config here',
}

# A user resource
user { 'nginx':
  ensure => present,
  uid    => 1000,
}

# File: /etc/puppetlabs/code/manifests/site.pp

# Define what all servers should have
node default {
  package { 'curl':
    ensure => installed,
  }
  
  package { 'wget':
    ensure => installed,
  }
  
  package { 'git':
    ensure => installed,
  }
}

# Define what web servers should have
node 'web1.company.com' {
  package { 'nginx':
    ensure => installed,
  }
  
  service { 'nginx':
    ensure => running,
  }
}

/etc/puppetlabs/code/modules/
├── nginx/                  # nginx module
│   ├── manifests/
│   │   ├── init.pp        # Main class
│   │   ├── install.pp     # Install logic
│   │   ├── config.pp      # Configuration logic
│   │   └── service.pp     # Service management
│   ├── templates/
│   │   └── nginx.conf.erb # Configuration template
│   └── files/
│       └── default.conf   # Default config file
├── mysql/                 # mysql module
│   ├── manifests/init.pp
│   └── templates/
└── users/                 # users module
    └── manifests/init.pp

node 'web1.company.com' {
  include nginx
  include mysql
  include users
}

Facts available on each server:
├─ os.name = 'Ubuntu'
├─ os.release.major = '20'
├─ os.family = 'Debian'
├─ networking.hostname = 'web1'
├─ networking.ip = '10.0.1.50'
├─ memory.system.total_bytes = '16000000000'
├─ processors.count = 8
└─ ... (100+ facts available)

# Install different packages based on OS
if $facts['os']['family'] == 'Debian' {
  package { 'nginx':
    ensure => installed,
  }
} elsif $facts['os']['family'] == 'RedHat' {
  package { 'nginx':
    ensure => installed,
  }
}

# Use different config based on OS
file { '/etc/nginx/nginx.conf':
  ensure  => file,
  content => $facts['os']['family'] ? {
    'Debian' => template('nginx/debian.conf.erb'),
    'RedHat' => template('nginx/redhat.conf.erb'),
  },
}

mkdir -p /etc/puppetlabs/code/modules/nginx/{manifests,templates,files}

modules/nginx/
├── manifests/
│   ├── init.pp          # Main class
│   ├── install.pp       # Installation
│   ├── config.pp        # Configuration
│   └── service.pp       # Service management
└── templates/
    └── nginx.conf.erb   # nginx configuration template

class nginx (
  String $version = 'latest',
  String $user = 'www-data',
  Integer $worker_processes = $facts['processors']['count'],
  Integer $worker_connections = 1024,
) {
  
  # Include other classes in order
  include nginx::install
  include nginx::config
  include nginx::service
  
  # Ensure config is deployed before service starts
  Class['nginx::install']
    -> Class['nginx::config']
    -> Class['nginx::service']
}

class nginx::install (
  String $version = $nginx::version,
) {
  
  # Update package cache (Debian/Ubuntu only)
  if $facts['os']['family'] == 'Debian' {
    exec { 'apt-update':
      command => 'apt-get update',
      path    => ['/usr/bin', '/usr/sbin'],
      onlyif  => 'test -f /etc/debian_version',
    }
  }
  
  # Install nginx package
  package { 'nginx':
    ensure  => $version,
    require => Exec['apt-update'],
  }
}

class nginx::config (
  String $user = $nginx::user,
  Integer $worker_processes = $nginx::worker_processes,
  Integer $worker_connections = $nginx::worker_connections,
) {
  
  # Deploy nginx configuration from template
  file { '/etc/nginx/nginx.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template('nginx/nginx.conf.erb'),
    notify  => Service['nginx'],
  }
  
  # Deploy default site config
  file { '/etc/nginx/sites-enabled/default':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => template('nginx/default-site.conf.erb'),
    notify  => Service['nginx'],
  }
  
  # Ensure sites-available directory exists
  file { '/etc/nginx/sites-available':
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
  }
  
  # Create log directory
  file { '/var/log/nginx':
    ensure  => directory,
    owner   => 'root',
    group   => 'root',
    mode    => '0755',
    require => Package['nginx'],
  }
}

class nginx::service {
  
  # Ensure nginx service is running and enabled on boot
  service { 'nginx':
    ensure    => running,
    enable    => true,
    subscribe => File['/etc/nginx/nginx.conf'],
  }
}

user <%= @user %>;
worker_processes <%= @worker_processes %>;
pid /run/nginx.pid;

events {
  worker_connections <%= @worker_connections %>;
}

http {
  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  gzip on;

  include /etc/nginx/sites-enabled/*;
}

# Apply default nginx to all servers
node default {
  class { 'nginx':
    version              => 'latest',
    worker_processes     => $facts['processors']['count'],
    worker_connections   => 2048,
  }
}

# Custom nginx config for high-traffic servers
node /^web-premium-.*/ {
  class { 'nginx':
    version              => 'latest',
    worker_processes     => $facts['processors']['count'],
    worker_connections   => 4096,  # More connections
  }
}

Monday 9:00 AM:
└─ Requirement: "All servers must run nginx 1.18"

Monday 9:15 AM:
└─ Update Puppet manifest:
   Change: version => 'latest'
   To:     version => '1.18'

Monday 9:30 AM:
└─ Puppet agents check in (every 30 min)
   ├─ Server 1: "Master says nginx 1.18, I have 1.16"
   │           → Apt update → Install 1.18 → Report done
   │
   ├─ Server 2: "Master says nginx 1.18, I have 1.18"
   │           → No change → Report done
   │
   ├─ Server 3-500: Same process
   │           → All servers update automatically
   │
   └─ Master dashboard: "500/500 agents in desired state"

Monday 10:30 AM:
└─ All 500 servers running nginx 1.18
    Zero manual work. Zero SSH logins. Problem solved.

Monday 9:00 AM:
└─ Requirement: "All servers must run nginx 1.18"

Monday 9:15 AM - Tuesday 2:00 PM:
├─ Create deployment script
├─ Test on 1 server
├─ Discover compatibility issue
├─ Fix script
├─ SSH to each server individually (or use parallel SSH)
├─ Run script on server 1-50 (works)
├─ Run script on server 51 (fails - different OS)
├─ Debug server 51
├─ Run script on server 52-100 (works)
├─ Run script on server 101 (hangs - old version conflict)
├─ Kill process, troubleshoot
├─ Manually fix server 101
├─ Run script on server 102-500
├─ Find 20 servers that failed
├─ Manually SSH to each, debug, fix
└─ Finally done: 40+ hours of work, lots of failed attempts

Result: 500 servers, 10 different versions running

Repository structure:
puppet-infrastructure/
├── modules/
│   ├── nginx/
│   ├── mysql/
│   └── users/
├── manifests/
│   ├── site.pp
│   └── nodes/
├── hieradata/        # Configuration data
└── .git/

Workflow:
1. Developer: Create feature branch
   git checkout -b add-postgres-module

2. Edit manifests in branch

3. Test in staging
   puppet agent -t --noop (show what would change, don't apply)

4. Create pull request

5. Code review (someone else approves)

6. Merge to main branch
   git merge add-postgres-module

7. Puppet master pulls latest
   Agents pick up changes automatically

hieradata/
├── common.yaml
├── os/
│   ├── debian.yaml
│   └── redhat.yaml
├── roles/
│   ├── webserver.yaml
│   ├── database.yaml
│   └── cache.yaml
└── nodes/
    ├── web1.company.com.yaml
    └── db1.company.com.yaml

nginx::version: '1.18'
nginx::worker_connections: 1024
mysql::version: '5.7'

nginx::package_name: 'nginx'
mysql::package_name: 'mysql-server'

nginx::worker_connections: 4096
php::version: '7.4'

nginx::listen_port: 8080  # Different port on this server

class nginx (
  String $version = lookup('nginx::version'),
  Integer $worker_connections = lookup('nginx::worker_connections'),
) {
  # ...
}

# On agent: Show what WOULD change (don't apply)
puppet agent -t --noop

# Returns:
# Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/nginx.conf]
# Notice: defined content as '{md5}abc123def456'
# Notice: These 3 resources would change

puppet agent -t
Error: Could not find a catalog from the master

# Check certificate
puppet cert list

# Sign unsigned certs
puppet cert sign agent-hostname

# Check agent connectivity
puppet config print server  # Shows master hostname

# Verify DNS resolves master
nslookup puppet.company.com

# Check time sync
timedatectl status

# Manually test connection
curl -v https://puppet.company.com:8140

You manually edit /etc/nginx/nginx.conf on server
Puppet agent runs after 30 min
Your changes are overwritten by Puppet config

Deploy new nginx config to 500 servers
3 servers crash due to config syntax error
Load balancer detects failure, stops routing traffic to those 3
Alerts fire
Incident war room starts

# Validate manifest syntax before deploying
puppet parser validate modules/nginx/manifests/*.pp

# Test on one server first
puppet agent -t --server puppet-test.company.com

# Use staged rollout
# Deploy to 5 test servers first
# Wait 1 hour, monitor
# Deploy to 50 servers
# Wait 1 hour, monitor
# Deploy to remaining 445 servers

# Rollback in Git (revert bad commit)
git revert abc123def

# Puppet master pulls latest
# Agents check in within 30 min
# Agents automatically revert to previous good config
# Production recovers automatically

  roles/ (combines modules for a job)
  ├── webserver.pp (nginx + php + ssl)
  ├── database.pp (mysql + backups + monitoring)
  └── cache.pp (redis + clustering)

  profiles/ (reusable configurations)
  ├── base_os.pp (common to all servers)
  ├── security.pp (firewall, selinux, etc.)
  └── monitoring.pp (all monitoring setup)

Terraform provisions servers (IaC)
├─ Creates 500 EC2 instances
├─ Creates load balancer
├─ Creates networking
└─ Outputs: Server IPs, hostnames

Puppet configures servers (Configuration Management)
├─ Connects to servers
├─ Installs packages
├─ Deploys configuration
├─ Starts services

# Terraform creates instance
resource "aws_instance" "web" {
  count           = 500
  ami             = "ubuntu-20.04"
  instance_type   = "t3.large"
  
  tags = {
    Name = "web-${count.index}"
    Role = "webserver"      # Puppet uses this tag
  }
}

# Puppet node definitions use tags
node /^web-.*/ {
  include roles::webserver
}

Puppet deploys monitoring agent
├─ Installs monitoring software
├─ Deploys configuration
├─ Connects to monitoring server
└─ Reports metrics

Monitoring dashboard shows
├─ What's installed on each server
├─ Configuration differences
├─ Compliance status

class monitoring_agent (
  String $server = 'prometheus.company.com',
  Integer $port = 9090,
) {
  package { 'prometheus-node-exporter':
    ensure => installed,
  }
  
  service { 'prometheus-node-exporter':
    ensure => running,
    enable => true,
  }
  
  file { '/etc/prometheus-node-exporter.conf':
    ensure  => file,
    content => template('monitoring/exporter.conf.erb'),
    notify  => Service['prometheus-node-exporter'],
  }
}

Manual configuration (500 servers):
├─ 2 hours per server = 1,000 hours
├─ Cost: €25,000-50,000 (labor)
├─ Inconsistency: HIGH
├─ Future changes: 1,000 hours per change

Puppet configuration:
├─ 40 hours to build module
├─ Cost: €1,000-2,000
├─ Inconsistency: ZERO (enforced by Puppet)
├─ Future changes: 30 minutes per change

Payoff: Breaks even after first change
Return: Massive time savings forever

✓ Consistency: 500 servers identical
✓ Repeatability: Deploy same config 1,000 times
✓ Safety: Rollback in minutes (use Git)
✓ Audit trail: Know who changed what when
✓ Scale: Manage thousands of servers
✓ Automation: Reduce manual work 90%+
✓ Compliance: Enforce security settings automatically

# On master server
wget https://apt.puppetlabs.com/puppet-release-focal.deb
dpkg -i puppet-release-focal.deb
apt-get update
apt-get install -y puppetserver

systemctl start puppetserver
systemctl enable puppetserver

# On all agent servers
wget https://apt.puppetlabs.com/puppet-release-focal.deb
dpkg -i puppet-release-focal.deb
apt-get update
apt-get install -y puppet-agent

# Configure agent to contact master
echo "server = puppet.company.com" >> /etc/puppetlabs/puppet/puppet.conf

systemctl start puppet
systemctl enable puppet

# On master: See unsigned certificates
puppet cert list

# Sign them
puppet cert sign agent1.company.com
puppet cert sign agent2.company.com

# Create site.pp
cat > /etc/puppetlabs/code/manifests/site.pp << 'EOF'
node default {
  package { 'curl':
    ensure => installed,
  }
}
EOF

# On agent
puppet agent -t

# Should show:
# Notice: Applied catalog in 1.23 seconds