Creating an Ubuntu 18.04 Server Foreman

Install

Just do a basic install using ubuntu-18.04.3-live-server-amd64.iso

Prevent Floppy Errors

If your server does not have a floppy, Ubuntu will report errors. This can be prevented by preventing the loading of a floppy driver on the clients:

echo "blacklist floppy" | sudo tee /etc/modprobe.d/blacklist-floppy.conf
sudo rmmod floppy
sudo update-initramfs -u

Enable Netowrking

The first time I built my VM, networking didn't install correctly. If this happens, create a netplan in /etc/netplans (note that because it's YAML, indentation is important)

network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: true

Apply the netplan

sudo netplan apply

Verify that it worked

ifconfig

Host Name Setup

Ensure the Hostname is Set Correctly

sudo hostnamectl set-hostname puppet.bpopp.net

Modify the /etc/hosts file:

127.0.0.1 localhost
192.168.1.42 puppet.bpopp.net

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Install Foreman

Install Puppet

sudo apt-get -y install ca-certificates
cd /tmp && wget https://apt.puppet.com/puppet6-release-bionic.deb
sudo dpkg -i /tmp/puppet6-release-bionic.deb

There appear to be issues with puppet 6 and newer versions of foreman. Puppet 5 may work better:

wget https://apt.puppetlabs.com/puppet5-release-xenial.deb
sudo dpkg -i puppet5-release-xenial.deb
sudo apt update

Enable Foreman

echo "deb http://deb.theforeman.org/ bionic 1.22" | sudo tee /etc/apt/sources.list.d/foreman.list
echo "deb http://deb.theforeman.org/ plugins 1.22" | sudo tee -a /etc/apt/sources.list.d/foreman.list
sudo apt-get -y install ca-certificates
wget -q https://deb.theforeman.org/pubkey.gpg -O- | sudo apt-key add -

Install

sudo apt-get update && sudo apt-get -y install foreman-installer

Run

sudo foreman-installer \
 --enable-foreman-proxy \
 --foreman-proxy-tftp=true \
 --foreman-proxy-tftp-servername=192.168.1.42 \
 --foreman-proxy-dhcp=true \
 --foreman-proxy-dhcp-gateway=192.168.1.1 \
 --foreman-proxy-dhcp-nameservers="192.168.1.1" \
 --foreman-proxy-dhcp-range="192.168.1.150 192.168.1.175" \
 --enable-foreman-compute-vmware \

To add DNS:

 --foreman-proxy-dns

Add the Puppetmaster as Host

Seems to be helpful to restart the puppetmaster after installing. To add the puppetmaster as a host, run this from the master:

sudo /opt/puppetlabs/bin/puppet agent --test

Now add our first module:

sudo /opt/puppetlabs/bin/puppet module install puppetlabs/ntp

Import the module from foreman:

• Click Configure > Classes
• Select "Any Organization" and "Any Location" from the menus in the menubar at the top
• Click the "Import environments from puppet" button
• Check the box next the module and click the "Update" button

Other useful modules

sudo /opt/puppetlabs/bin/puppet module install puppetlabs/motd
sudo /opt/puppetlabs/bin/puppet module install puppetlabs/accounts

/etc/puppetlabs/code/environments/production/modules
├── adrien-alternatives (v0.3.0)
├── camptocamp-kmod (v2.2.0)
├── cesnet-autoupdate (v1.0.2)
├── cesnet-hadoop (v0.9.6)
├── cesnet-spark (v0.2.0)
├── puppet-archive (v3.2.1)
├── puppet-nginx (v1.0.0)
├── puppet-php (v6.0.2)
├── puppet-zypprepo (v2.2.2)
├── puppetlabs-accounts (v4.2.0)
├── puppetlabs-apt (v6.3.0)
├── puppetlabs-concat (v6.1.0)
├── puppetlabs-git (v0.5.0)
├── puppetlabs-inifile (v2.4.0)
├── puppetlabs-kubernetes (v5.0.0)
├── puppetlabs-motd (v3.1.0)
├── puppetlabs-ntp (v8.0.0)
├── puppetlabs-registry (v2.1.0)
├── puppetlabs-stdlib (v6.0.0)  invalid
├── puppetlabs-translate (v2.0.0)  invalid
└── saz-ssh (v1.0.3)


Error: Could not uninstall module 'puppetlabs-stdlib'
  Other installed modules have dependencies on 'puppetlabs-stdlib' (v6.0.0)
    'cesnet/autoupdate' (v1.0.2) requires 'puppetlabs-stdlib' (>= 1.0.0 < 5.0.0)
    'cesnet/hadoop' (v0.9.6) requires 'puppetlabs-stdlib' (>= 1.0.0)
    'cesnet/spark' (v0.2.0) requires 'puppetlabs-stdlib' (>= 1.0.0)
    'puppet/archive' (v3.2.1) requires 'puppetlabs-stdlib' (>= 4.13.1 < 6.0.0)
    'puppet/nginx' (v1.0.0) requires 'puppetlabs-stdlib' (>= 5.0.0 < 7.0.0)
    'puppet/php' (v6.0.2) requires 'puppetlabs-stdlib' (>= 4.16.0 < 6.0.0)
    'puppetlabs/accounts' (v4.2.0) requires 'puppetlabs-stdlib' (>= 5.0.0 < 7.0.0)
    'puppetlabs/apt' (v6.3.0) requires 'puppetlabs-stdlib' (>= 4.16.0 < 6.0.0)
    'puppetlabs/concat' (v6.1.0) requires 'puppetlabs-stdlib' (>= 4.13.1 < 7.0.0)
    'puppetlabs/git' (v0.5.0) requires 'puppetlabs-stdlib' (>= 3.2.0)
    'puppetlabs/kubernetes' (v5.0.0) requires 'puppetlabs-stdlib' (>= 4.20.0 <= 5.2.0)
    'puppetlabs/motd' (v3.1.0) requires 'puppetlabs-stdlib' (>= 2.1.0 < 7.0.0)
    'puppetlabs/ntp' (v8.0.0) requires 'puppetlabs-stdlib' (>= 4.13.1 < 7.0.0)
    Use `puppet module uninstall --force` to uninstall this module anyway

Configure Foreman

• Click Infrastructure > Subnets > Create Subnet
• Use the following settings:
  • Name: bpopp.net
  • Network address: 192.168.1.0
  • Network Prefix: 24
  • Network Mask: 255.255.255.0
  • Gateway Address: 192.168.1.1
  • Primary DNS: 192.168.1.1
  • IPAM: DHCP
  • Start of Ip Range: 192.168.1.150
  • End of Ip Range: 192.168.1.175
  • Boot Mode : DHCP
• In the tabs:
  • Under Domain tab, select bpopp.net
  • Under Proxies tab, select the puppet.bpopp.net proxies for DHCP, TFTP, and HTTPBoot
  • From Locations and Organizations, select Default Location/Organization

Configure Operating Systems

For a CentOS box:

• Click Hosts > Operating Systems > Create Operating System
• Use the following settings:
  • Name: CentOS
  • Major Version: 7
  • Family: RedHat
  • Architecture: x86_64
• From the tabs
  • In the partition table, select 'Kickstart default'
  • In the installation Media, select 'CentOS Mirror'

For a Debian box:

• Click Hosts > Operating Systems > Create Operating System
• Use the following settings:
  • Name: CentOS
  • Major Version: 7
  • Family: Debian
  • Release Name: stretch (important!)
  • Architecture: x86_64
• From the tabs
  • In the partition table, select 'Preseed default'
  • In the installation Media, select 'Debian Mirror'

Assign Partition Tables

This part of the configuration seemed really goofy, but is important. You have to assign your provisioning templates to any operating systems you want them to be available from.

For CentOS:

• Click Hosts > Provisioning Templates
• Type "kickstart" from the search box
• Assign each of the following templates to CentOS by clicking their "Association" tab and selecting CentOS. Note that you cannot change any other setting. If you would like to change anything, you have to clone them and then assign the clone to your OS.
  • Kickstart Default
  • Kickstart default finish
  • Kikstart default iPXE
  • Kickstart default PXELinux
  • Kickstart default user data
• For Debian, change the following:
  • Preseed default
  • Preseed default finish
  • Preseed default iPXE
  • Preseed default PXELinux
  • Preseed default user data

After assigning each template to the correct operating systems:

• Click Hosts > Operating System
• Click each operating system and click the "Templates" tab
• Assign any desired provisioning templates to the operating systems (typically all 5)

Create a Host Group

Host groups are used to quickly provision/configure servers of a specific type. For example, you can define a host group for Apache webservers that have all the correct modules installed for that specific type of server.

With all the previous configurations complete:

• Click Configure > Host groups > Create Host Group
• Use the following settings:
  • Name: Basic
  • Environment: production
  • Puppet Master: puppet.bpopp.net
  • Puppet CA: puppet.bpopp.net
• From the tabs:
  • From Puppet Classes, add any desired modules for this host group
  • From Network
    • Domain: bpopp.net
    • IPv4: bpopp.net
  • From Operating System
    • Architecture: x86_64
    • Operating System: CentOS
    • Media: CentOS Mirror
    • PXELoader: PXELinux BIOS
    • Root Password: whatever
  • From Locations and Organizations, select Default Location/Organization

Create First Host

• Click Hosts > Create Host
• Use settings:
  • Name: client1
  • Organization: Default
  • Location: Default
  • Host Group: basic

After selecting the correct host group, each of the tabs should fill in automatically with correct information for that host group. You will, however, need to assign the interface to a specific box. There's probably a better way to do this, but what works is:

• Create a VM in ESXI
• Start it briefly to generate a Mac Address
• Stop it
• Look in the settings and copy the generated MAC address to foreman

Manifests

Took me way too long to figure this out, but after installing the accounts plugin from puppetforge, I couldn't really figure out how to configure it. Turns out it is actually really simple. There's undoubtedly multiple ways to do this, but any configuration files added /etc/puppetlabs/code/environment/production/manifests will automatically get applied to production boxes. So for example, to add a user, add the following to users.pp:

package  { [ "cron",
             "locate",
             "lsof",
             "screen",
             "sudo",
             "unzip" ]:
     ensure => installed,
}


accounts::user { 'bpopp':
  group    => 'bpopp',
  groups   => [ 'sudo' ],
  shell    => '/bin/bash',
  password => 'encoded',
  locked   => false,
}

To create the password, you can use:

mkpasswd -m sha-512

(may require installing apt install whois)

What I really want to be able to do is assign this script only to a specific host group, but this is a good start.

Set the Timezone

Install the timezone module

puppet module install saz-timezone

Create a timezone manifest file @ /etc/puppetlabs/code/environments/production/manifests/timezone.pp

class { 'timezone':
   timezone => 'America/Chicago',
}

Docker Manifest File

/etc/puppetlabs/code/environments/production/manifests/docker.pp

node /^docker\d+\.bpopp\.net/
{
        include 'docker'
        docker::image { 'jupyter/pyspark-notebook': }

docker::run { 'helloworld':
  ensure => absent,
  image => 'ubuntu',
}

        docker::run { 'pyspark-notebook':
          image   => 'jupyter/pyspark-notebook',
          ports => ['8888'],
          extra_parameters => [ '--it ' ],
        }
}

Packages

/etc/puppetlabs/code/environments/production/manifests/packages.pp

package { 'curl':
   ensure => present,
}

Users

To manage keys, you'll need to install the sshkeys_core module:

sudo puppet module install puppetlabs-sshkeys_core --version 2.4.0

Then:

accounts::user { 'bpopp':
  group    => 'bpopp',
#  groups   => [ 'sudo' ],
  shell    => '/bin/bash',
  password => '$encoded',
  locked   => false,

sshkeys => [
    'ssh-rsa AAAAB3NzaCcensoredw== bpopp@bpopp.net',
  ],
}

class { 'sudo':
  config_file_replace => false,
}

sudo::conf { 'bpopp':
  priority => 60,
  content  => "bpopp ALL=(ALL) NOPASSWD: ALL",
}

Don't remember what this is for?

sudo::conf { 'foreman-proxy':
  priority => 60,
  content  => "foreman-proxy ALL = NOPASSWD: /opt/puppetlabs/bin/puppet cert *",
}

Spark Manifest

Rather elaborate script to install Spark and hadoop

node /^(spark\d+|dataiku)\.lab.bpopp\.net/
{
   $spark_install_path = "/usr/local/spark"
   $hadoop_install_path = "/usr/local/hadoop"
   $livy_install_path = "/usr/local/livy"
   $spark_filename = "spark-2.4.5.tgz"
   $hadoop_filename = "hadoop-3.2.1.tar.gz"
   $livy_filename = "apache-livy-0.6.0-incubating-bin.zip"
   $sparkuser = "spark"

   package { 'scala':
        ensure => 'installed',
   }

   package { 'python3':
        ensure => 'installed',
   }

   package { 'python':
        ensure => 'installed',
   }


   package { 'python3-pip':
        ensure => 'installed',
   }

   package { 'bsdtar':
        ensure => 'installed',
   }

   accounts::user { $sparkuser:
     group    => $sparkuser,
     shell    => '/bin/bash',
     password => '$6$censored/f/0HTyufJDHIc.0a/pT0vQ0S1',
     locked   => false,
   }


   include 'archive' # NOTE: optional for posix platforms

   file { $spark_install_path:
     ensure => directory,
     owner  => $sparkuser,
     group  => $sparkuser,
     mode   => '0755',
   }

  file { $livy_install_path:
     ensure => directory,
     owner  => $sparkuser,
     group  => $sparkuser,
     mode   => '0755',
   }

   file { $hadoop_install_path:
     ensure => directory,
     owner  => $sparkuser,
     group  => $sparkuser,
     mode   => '0755',
   }

   archive { "/tmp/${spark_filename}":
     ensure        => present,
     extract       => true,
     extract_path  => $spark_install_path,
     extract_command => 'tar xfz %s --strip-components=1',
     source        => "https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-without-hadoop.tgz",
     creates       => "${spark_install_path}/bin",
     cleanup       => true,
     require       => File[$spark_install_path],
   }

   archive { "/tmp/${hadoop_filename}":
     ensure        => present,
     extract       => true,
     extract_path  => $hadoop_install_path,
     extract_command => 'tar xfz %s --strip-components=1',
     source        => "https://archive.apache.org/dist/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz",
     creates       => "${hadoop_install_path}/bin",
     cleanup       => true,
     require       => File[$hadoop_install_path],
   }

 archive { "/tmp/${livy_filename}":
     ensure        => present,
     extract       => true,
     extract_path  => $livy_install_path,
     extract_command => 'bsdtar xfz %s --strip-components=1',
     source        => "http://mirrors.ibiblio.org/apache/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip",
     creates       => "${livy_install_path}/bin",
     cleanup       => true,
     require       => File[$livy_install_path],
   }


   file { '/etc/profile.d/spark.sh':
       mode    => '644',
       content => 'PATH=$PATH:/usr/local/spark/bin:/usr/local/hadoop/bin; JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/; PYSPARK_PYTHON=/usr/bin/python; HADOOP_HOME=/usr/local/hadoop; SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath); SPARK_HOME=/usr/local/spark;',
   }

   file { "/usr/local/spark/conf/spark-env.sh":
    ensure => present,
   }

   file_line { "/usr/local/spark/conf/spark-env.sh":
    path => "/usr/local/spark/conf/spark-env.sh",
    line => "export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)",
   }

   file_line { "/usr/local/hadoop/etc/hadoop/hadoop-env.sh":
    path => "/usr/local/hadoop/etc/hadoop/hadoop-env.sh",
    line => "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/",
   }

   file { "/usr/local/livy/conf/livy-env.sh":
    ensure => present,
   }

   file_line { "/usr/local/livy/conf/livy-env.sh":
    path => "/usr/local/livy/conf/livy-env.sh",
    line => "export SPARK_HOME=/usr/local/spark",
   }

}

Hadoop Manifest

class{"hadoop":
  hdfs_hostname => 'spark-master.bpopp.net',
  yarn_hostname => 'spark-master.bpopp.net',
  slaves => [ 'spark-master.bpopp.net' ],
  frontends => [ 'spark-master.bpopp.net' ],
  properties => {
    'dfs.replication' => 1,
  },
  realm => '',
  features => {},
}


node 'spark-master.bpopp.net' {
  include hadoop

 # HDFS
  include hadoop::namenode
  # YARN
  include hadoop::resourcemanager
  # MAPRED
  include hadoop::historyserver
  # slave (HDFS)
  include hadoop::datanode
  # slave (YARN)
  include hadoop::nodemanager
  # client
  include hadoop::frontend

}

node default {
}

Resetting Kernel/Boot

Foreman downloads kernel/boot files to /var/lib/tftpboot/boot. If installing newer version of the same OS, these files are cached and could return an error that says, "No kernel modules were found.". If this happens, purge this folder and they will be rebuild.

From the documentation:

The kernel and initrd are not deleted, but left in place for future installs of the same OS and architecture combination. Please note that in the unlikely case that these files are modified, the simplistic freshness check of wget will likely get confused, corrupting the downloaded versions of the files. If this happens, you should simply delete the files and let them be re-downloaded from scratch.