The Pragmatic Sysadmin

  • Tired of looking for all pieces of the puzzle yourself?

  • Tired of fighting through an endless djungle of (obsolete) HowTos and Information
    just to discover that you only needed a small piece of all that information
    maybe because most of it is written for big infrastructures like Facebook
    instead for a common environment?

  • Tired of trying out every tool yourself
    and just want to know which are the tried-and-true ones?

Then maybe this book could be for you.

  • Pragmatic solutions, tips, tricks and tools for common SysAdmin work
  • Written from the Admin for the Admin

During my career I wished there was such a book.
There was none, so I write it myself:

a) to use it as my own reference
b) to give back what I learned from others and my own experience

Target Audience

  • Basic knowledge with the CLI and some SysAdmin experience

What is a sysadmin ?

"Aside from the rigors of setting up all the usual software and hardware,
your primary job as a sysadmin is to find solutions.

There will be times when you encounter a problem outside your job description,
and it may not even be possible for you to fix it, but it'll be up to you to find a workaround."

https://opensource.com/article/19/7/be-a-sysadmin

What is a pragmatic one ?

Setting up and maintaining all these solutions
in the most efficient (hopefully also elegant) way
with minimal overhead (progressive enhancement design principle)
so that you have time to improve and learn the other new stuff

-- swick

progressive enhancement design principle:

The "progressive enhancement" design principle
requires minimal functionality using what is available
and progressively upgrading based on what is detected

-- rwxrob

Debian Packaging

  • There are countless (often deprecated) tutorials
    and different variants about how to create a Debian package

  • Quite a difficult djungle to fight through even with a Machete
    only to discover that from all that information you only need a small amount.

  • Most HowTos start with Getting the Upstream Tarball
    but most of the time you don't have or need this.

  • Very often you only have a shell skript, binary or JAR file or something similar
    which you just want to get into a package
    so that you can take advantage of the system's package manager and not bypass it.

  • What follows is a guide to quickly create packages with some optional variants.

  • You should always make sure, that a package is in good shape, but applying all the rules
    of the Debian Policy would be overkill (Remember: We are pragmatic here)

    If you want to get your package into Debian,
    then of course you should follow their advice and policies...

Creating

There are many ways to create Debian packages.

I show you the standard way with some variants which are also pragmatic.

The Standard Way

Install necessary tools

apt-get install build-essential debhelper devscripts

List files

find ~/packaging/foo/ -type f | sort


/home/swick/packaging/foo/debian/changelog
/home/swick/packaging/foo/debian/control
/home/swick/packaging/foo/debian/install
/home/swick/packaging/foo/debian/rules
/home/swick/packaging/foo/foo

~/packaging/foo/foo

#!/usr/bin/env bash
  
echo "foo"

~/packaging/foo/debian/control

Source: foo
Section: utils
Priority: optional
Maintainer: Sven Wick <sven.wick@gmx.de>
Build-Depends: debhelper-compat (= 12)
Standards-Version: 4.6.0
Homepage: https://github.com/foo/
 
Package: foo
Architecture: all
Depends: ${misc:Depends}
Description: short description for foo
 Longer description with more information
 .
 and a second paragraph

~/packaging/foo/debian/rules

#!/usr/bin/make -f
 
%:
    dh $@
  • The rules file is a Makefile and MUST be executable ( chmod +x )
  • The line with dh MUST be indented with a TAB (not spaces)

~/packaging/foo/debian/install

foo    usr/bin

~/packaging/foo/debian/changelog

foo (0.1-1) unstable; urgency=medium

  * Initial release

 -- Sven Wick <sven.wick@gmx.de>  Fri, 17 Dec 2021 00:29:27 +0100


cd ~/packaging/foo
dpkg-buildpackage -uc -us

dpkg-deb -c ~/packaging/foo_0.1-1_all.deb

drwxr-xr-x root/root         0 2021-12-17 00:29 ./
drwxr-xr-x root/root         0 2021-12-17 00:29 ./usr/
drwxr-xr-x root/root         0 2021-12-17 00:29 ./usr/bin/
-rwxr-xr-x root/root        34 2021-12-17 00:29 ./usr/bin/foo
drwxr-xr-x root/root         0 2021-12-17 00:29 ./usr/share/
drwxr-xr-x root/root         0 2021-12-17 00:29 ./usr/share/doc/
drwxr-xr-x root/root         0 2021-12-17 00:29 ./usr/share/doc/foo/
-rw-r--r-- root/root       134 2021-12-17 00:29 ./usr/share/doc/foo/changelog.Debian.gz


dpkg-deb -I ~/packaging/foo_0.1-1_all.deb

 new Debian package, version 2.0.
 size 1172 bytes: control archive=524 bytes.
     281 bytes,    12 lines      control              
     118 bytes,     2 lines      md5sums              
 Package: foo
 Version: 0.1-1
 Architecture: all
 Maintainer: Sven Wick <sven.wick@gmx.de>
 Installed-Size: 9
 Section: utils
 Priority: optional
 Homepage: https://github.com/foo/
 Description: short description for foo
  Longer description with more information
  .
  and a second paragraph

Create a little helper

Let's create a little script which creates the minimum of files we need for a package.

Works well if you just want to package a bunch of files

mkdir -p ~/packaging/mkdeb/

~/packaging/mkdeb/mkdeb

#!/usr/bin/env bash

#
# Usage
#

if [[ $1 == "-h" ]] || [[ $1 == "--help" ]]; then
  echo
  echo "  mkdeb [package_name] (default: basename of \$PWD)"
  echo
  exit
fi

TAB="$(printf '\t')"

#
# Package name
#

if [[ -z ${1} ]]; then
  PKG=$(basename ${PWD})
else
  PKG=${1}
fi

#
# Email
#

if [[ ${DEBEMAIL} == *@* ]]; then
  EMAIL=${DEBEMAIL}
else
  EMAIL="${USER}@${HOSTNAME}"
fi

#
# Fullname
#

NAME=$(getent passwd ${USER} | awk -F: '{print $5}')

if [[ ${NAME} =~ [0-9a-zA-Z] ]]; then
  FULL_NAME=${NAME}
else
  FULL_NAME=${USER}
fi

#
# debian/
#

if [[ -d debian ]]; then
  echo "There is already a debian folder"
  exit
fi

echo
echo "Creating debian/"
mkdir debian

#
# debian/rules
#
   
echo "Creating debian/rules"

cat << EOF > debian/rules
#!/usr/bin/make -f

%:
${TAB}dh \$@
EOF

chmod +x debian/rules

#
# debian/install
#

echo "Creating debian/install"

cat << EOF > debian/install
${PKG} usr/bin
EOF

#
# debian/changelog
#

echo "Creating debian/changelog"

cat << EOF > debian/changelog
${PKG} (0.1-1) unstable; urgency=medium

  * Initial release

 -- ${FULL_NAME} <${EMAIL}>  $(date "+%a, %d %b %Y %T %z")
EOF

#
# debian/control
#

echo "Creating debian/control"

cat << EOF > debian/control
Source: ${PKG}
Section: utils
Priority: optional
Maintainer: ${FULL_NAME} <${EMAIL}>
Build-Depends: debhelper-compat (= 12)
Standards-Version: 4.6.0
Homepage: https://${PKG}.de

Package: ${PKG}
Architecture: all
Depends: \${misc:Depends}
Description: short description of ${PKG}
 Long description of ${PKG}

EOF

You could use this shell script as is
but of course we make a package out of it

~/packaging/mkdeb/debian/control

Source: mkdeb
Section: utils
Priority: optional
Maintainer: Sven Wick <sven.wick@gmx.de>
Build-Depends: debhelper-compat (= 12)
Standards-Version: 4.6.0
Homepage: https://vaporup.github.io/books/the-pragmatic-sysadmin

Package: mkdeb
Architecture: all
Depends: bash
Description: quickly create the files for a deb package
 mkdeb creates the minimum of files
 to quickly prepare a Debian package

~/packaging/mkdeb/debian/install

mkdeb usr/bin

~/packaging/mkdeb/debian/changelog

mkdeb (0.1-1) unstable; urgency=medium

  * Initial release

 -- Sven Wick <sven.wick@gmx.de>  Sun, 12 Dec 2021 02:14:45 +0100

~/packaging/mkdeb/debian/rules

#!/usr/bin/make -f

%:
    dh $@
  • The rules file is a Makefile and MUST be executable ( chmod +x )
  • The line with dh MUST be indented with a TAB (not spaces)
cd ~/packaging/mkdeb
dpkg-buildpackage -uc -us

Variants

Now some variants if for some reason the standard way is not possible.

Variant 1

apt-get install equivs


mkdir -p ~/packaging/foo/

~/packaging/foo/foo

#!/usr/bin/env bash

echo "foo"

~/packaging/foo/control

Package: foo
Version: 0.1-1
Architecture: all
Maintainer: Sven Wick <sven.wick@gmx.de>
Homepage: https://github.com/foo/
Files: foo usr/bin
Description: short description for foo

equivs-build ~/packaging/foo/control

dpkg-deb -c foo_0.1-1_all.deb

drwxr-xr-x root/root         0 2021-12-17 01:21 ./
drwxr-xr-x root/root         0 2021-12-17 01:21 ./usr/
drwxr-xr-x root/root         0 2021-12-17 01:21 ./usr/bin/
-rwxr-xr-x root/root        34 2021-12-17 01:21 ./usr/bin/foo
drwxr-xr-x root/root         0 2021-12-17 01:21 ./usr/share/
drwxr-xr-x root/root         0 2021-12-17 01:21 ./usr/share/doc/
drwxr-xr-x root/root         0 2021-12-17 01:21 ./usr/share/doc/foo/
-rw-r--r-- root/root       741 2021-12-17 01:21 ./usr/share/doc/foo/README.Debian
-rw-r--r-- root/root       131 2021-12-17 01:21 ./usr/share/doc/foo/changelog.Debian.gz
-rw-r--r-- root/root       936 2021-12-17 01:21 ./usr/share/doc/foo/copyright

dpkg-deb -I foo_0.1-1_all.deb

 new Debian package, version 2.0.
 size 2176 bytes: control archive=556 bytes.
     232 bytes,    10 lines      control              
     246 bytes,     4 lines      md5sums              
 Package: foo
 Version: 0.1-1
 Architecture: all
 Maintainer: Sven Wick <sven.wick@gmx.de>
 Installed-Size: 11
 Section: misc
 Priority: optional
 Multi-Arch: foreign
 Homepage: https://github.com/foo/
 Description: short description for foo

Variant 2

This variant uses the same package format as Arch Linux,
so your source package very likely also works for Arch-based Distros

mkdir -p ~/packaging/foo/

~/packaging/foo/foo

#!/usr/bin/env bash

echo "foo"


chmod +x ~/packaging/foo/foo

~/packaging/foo/PKGBUILD

# Maintainer: Sven Wick <sven.wick@gmx.de>
 
pkgname=foo
pkgver=0.1
pkgrel=1
pkgdesc="short description for foo"
arch=("any")
url="https://github.com/foo/"
depends=("bash")
changelog=
 
package() {
 
    mkdir -p    $pkgdir/usr/bin/
    cp ../foo   $pkgdir/usr/bin/
}


cd ~/packaging/foo/
makedeb

dpkg-deb -c foo_0.1-1_all.deb

drwxr-xr-x root/root         0 2021-12-17 01:33 ./usr/
drwxr-xr-x root/root         0 2021-12-17 01:33 ./usr/bin/
-rw-r--r-- root/root        34 2021-12-17 01:33 ./usr/bin/foo

dpkg-deb -I foo_0.1-1_all.deb

 new Debian package, version 2.0.
 size 650 bytes: control archive=270 bytes.
     193 bytes,     8 lines      control              
 Package: foo
 Version: 0.1-1
 Description: short description for foo
 Architecture: all
 Maintainer: Sven Wick <sven.wick@gmx.de>
 Homepage: https://github.com/foo/
 Installed-Size: 14
 Depends: bash

More Variants

With minimal Tools

  • This is the most basic variant with minimal tools involved (quick-n-dirty).
  • You can go even more low-level with ar but that is not pragmatic anymore...
  • This variant should only be used if all other variants are not possible...

Install necessary tools

apt-get install fakeroot

fakeroot is needed so that the files in the package are owned by "root"
and not by the user who builds the package.

root (UID 0) exists on every linux system, but the current user's ID very likely does not

mkdir -p ~/packaging/foo/DEBIAN

This is the only time the DEBIAN folder needs to be in uppercase
since we use some lower level tools which still expect this...

Tools like debhelper and others use the lowercase variant.

mkdir -p ~/packaging/foo/usr/bin/

~/packaging/foo/usr/bin/foo

#!/usr/bin/env bash
  
echo "foo"

chmod +x ~/packaging/foo/usr/bin/foo

~/packaging/foo/DEBIAN/control

Package: foo
Version: 0.1-1
Architecture: all
Maintainer: Sven Wick <sven.wick@gmx.de>
Description: short description for foo

fakeroot dpkg-deb --build ~/packaging/foo/

dpkg-deb -c ~/packaging/foo.deb

drwxr-xr-x root/root         0 2021-12-17 01:01 ./
drwxr-xr-x root/root         0 2021-12-17 01:01 ./usr/
drwxr-xr-x root/root         0 2021-12-17 01:01 ./usr/bin/
-rwxr-xr-x root/root        34 2021-12-17 01:01 ./usr/bin/foo

dpkg-deb -I ~/packaging/foo.deb

 new Debian package, version 2.0.
 size 740 bytes: control archive=304 bytes.
     126 bytes,     5 lines      control              
 Package: foo
 Version: 0.1-1
 Architecture: all
 Maintainer: Sven Wick <sven.wick@gmx.de>
 Description: short description for foo

Testing

To make sure your packages are in good shape
there are some ways to test them:

Hosting

To fully integrate your packages with your system's package manager
provide an APT repo:

Providing an APT repo with aptly and Apache

To see an working example of this setup:

Install aptly

apt-get install aptly

Create an extra user

adduser dhl

Full Name []: User for APT repo using aptly

Make sure there is enough disk space for all the files
Eventually put the home of this user on a separate mount

Create a GPG Key

Creating a GPG key which signs the files in the APT repo
so apt clients do not complain

Using ed25519 as SSH Key type here, but other types (RSA), work as well

  1. Switch to your new user

  2. Create the key

    dhl@host: gpg --expert --full-generate-key

    (9) ECC and ECC

(1) Curve 25519

0 = key does not expire

Real name: APT REPO
Email address: dhl@your-domain.com

skip password

  3. Check the new GPG key

    gpg --list-secret-keys --keyid-format=long

  4. Export the new GPG key

    gpg --armor --export dhl@your-domain.com > dhl-repo-keyring.asc
gpg         --export dhl@your-domain.com > dhl-repo-keyring.gpg

Prepare aptly

Example for Ubuntu 20.04 (Focal)
Remember: Do everything as your new aptly user (here dhl)

  1. Create repo

    aptly repo create -distribution=focal -component=main focal-main

  2. Import some example package

    Can be a .deb (binary package) or .dsc (source package)

    aptly repo add focal-main example_1.14.1_amd64.deb
aptly repo add focal-main example_1.14.1.dsc

  3. Publish your packages

    Importing a package does not make it available for apt yet.
    It is available only after the repo was published.
    The following step is only done once to initialize the repo.

    For origin and label use some unique string which identifiers your repo

    aptly publish repo \
-architectures=amd64,i386,source \
-origin=some-string-which-identifies-your-repo \
-label=some-string-which-identifies-your-repo
focal-main

    If you import more packages, you do:

    aptly publish update focal

Providing an APT repo with aptly and Apache

/etc/apache2/sites-available/apt.your-domain.de.conf

<VirtualHost *:80>
  ServerName    apt.your-domain.de
  DocumentRoot  /home/dhl/.aptly/public/

  <Directory /home/dhl/.aptly/public/>
    IndexIgnore HEADER.html
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
  </Directory>

</VirtualHost>

Activate

root@host: a2ensite apt.your-domain.de.conf
root@host: systemctl reload apache2

Package Overview

/home/dhl/.aptly/public/HEADER.html

/home/dhl/.aptly/public/packages/index.html

/home/dhl/bin/create-package-list.py

Upload Facility

(Server) SFTP Chroot for Incoming

root@host: mkdir -p /home/dhl/upload/incoming
root@host: chown dhl:dhl /home/dhl/upload/incoming
root@host: chown root:root /home/dhl

/etc/ssh/sshd_config

Match user dhl
    ChrootDirectory /home/dhl/upload
    #AuthorizedKeysFile  /home/dhl/.ssh/authorized_keys
    ForceCommand internal-sftp
    AllowTCPForwarding no
    X11Forwarding no

/home/dhl/bin/import-packages.sh

(Client) dput

Used on the PC who wants to upload packages

apt-get install dput

~/.dput.cf

[your-server-focal]
fqdn = apt.your-server.de
# With different SSH port
#fqdn = apt.your-server.de:35007
incoming = /incoming/focal
method = sftp
login = dhl
allow_unsigned_uploads = 1
progress_indicator = 2
# Allow uploads for UNRELEASED packages
allowed_distributions = .*

Now you can upload packages with dput

dput your-server-focal example_0.1-1.changes

(Client) Optional

You can also just upload a .deb file with FileZilla or similar tools
into the appropriate incoming folder

Automate with Cronjob

crontab of dhl

Bonus

/etc/apache2/sites-available/packages.your-domain.de.conf

<VirtualHost *:80>

  ServerName   packages.your-domain.de
  DocumentRoot /home/dhl/.aptly/public/packages/

  <Directory /home/dhl/.aptly/public/packages/>
    AllowOverride None
    Require all granted
  </Directory>

</VirtualHost>


a2ensite packages.your-domain.de.conf
systemctl reload apache2

Update Packages

Debian/Ubuntu

apt-get update
apt-get upgrade
apt-get dist-upgrade

NetBSD

pkgin upgrade

FreeBSD

pkg upgrade

OpenBSD

pkg_add -uvi

sysupgrade

DragonFlyBSD

pkg upgrade

Arch/Manjaro

pacman -Sy archlinux-keyring
pacman -Syu

Fedora/CentOS/Alma

dnf upgrade

Gentoo

emerge --sync

Alpine

apk upgrade

OpenSUSE

zypper refresh
zypper update

void

xbps-install -Su

Slackware

slackpkg update
slackpkg install-new
slackpkg upgrade-all
slackpkg clean-system

Guix

guix pull
guix install htop

nixOS

nixos-rebuild switch --upgrade

Low-Budget Weiterbildung

Tools