Admin Scripts

This repo contains some administration scripts for administrering a debian machine. The covered tasks range from file change tracking via http/ssh monitoring to zfs health checking.

TODO: Add better description about host system and notification method in first sentence.

Installation

This repository can be checked out wherever you like. Typically this is ~/admin-scripts. The rest of this README assumes you checked out this repo into /root/admin-scripts.

cd
git clone ssh://git@git.mneun.ch:2222/radioelephant/admin-scripts.git

If you are unable to clone via SSH, try via https. SSH is preferrable.

cd
git clone https://git.mneun.ch/radioelephant/admin-scripts.git

You might need to install some utilities to be able to use the scripts in this repository. To install all required dependencies, use the following command:

sudo apt install moreutils bc lm-sensors
sudo sensors-detect --auto

To make sure a script is runnable, always run it manually first, before setting up cronjobs. If the script produces any unexpected output, you might be missing utilities or the configuration.

Please also quickly read the CHANGELOG and confirm this by running date > /root/scripts/.last_changelog_read. This will be helpful when updating the admin scripts later on.

Config Files

Config files are located in the config/ directory. Each script has its own cofiguration file, in this repo there only are the config/[script_name].EXAMPLE example configuration files. For each script you want to use, you must copy the example and fill in your own data. For example (monitoring.sh):

cd ~/admin-scripts
cp config/monitoring.EXAMPLE config/monitoring
vim config/monitoring

If you want to use the provided example configuration, you could symbolic link the files instead of just copying them. This really only makes sense for system_health_check and state_machine. The command for this is:

cd ~/admin-scripts
ln -s config/system_health_check.EXAMPLE config/system_health_check

Shutdown Notification

To be notified immediately before sever shutdown the contained run-before-shutdown.service can be installed as follows. This service file will send a telegram notification informing about the pending shutdown.

ln -s /root/admin-scripts/run-before-shutdown.service /etc/systemd/system/shutdown.target.wants/run-before-shutdown.service
systemctl daemon-reload

Adapt the above command, if your scripts are located somewhere else!

Startup Helper

To install the startup helper script into the regular user home use the following commands, while supplying your username on the first line.

USRNAME=radioelephant
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
cp /root/admin-scripts/config/post_startup.EXAMPLE /home/$USRNAME/.post_startup_env
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/.post_startup_env
vim /home/$USRNAME/.post_startup_env

Just shutdown your computer if you don't know how to exit VIM at this point.

See also Updating Startup Helper.

Updating

You can update the admin scripts by pulling from the git remote.

cd ~/admin-scripts
git pull

For most of the scripts you only need to check if the example configuration file (i.e. config/monitoring.EXAMPLE) has changed and contains different keys than your copy (i.e. config/monitoring). For your convenience, changes to configuration files will be documented in the CHANGELOG. If you followed the instructions in this README, then you will find the last time you pulled this repository and read the CHANGELOG in the .last_changelog_read file. Get the value with cat /root/admin-scripts/.last_changelog_read.

Make sure to update the last reading time file after reading the CHANGELOG with date > /root/scripts/.last_changelog_read

Updating Startup Helper

The startup helper script in the regular user home cannot be updated via git pull and needs to be compared and/or copied manually. To compare use the following commands:

USRNAME=radioelephant
diff /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
diff /root/admin-scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env

To copy the startup helper script again, use the following commands (first line not needed, if you just ran the last set of commands). Please adapt the environment file /home/[USRNAME]/.post_startup_env manually.

USRNAME=radioelephant
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh

Make sure to update the last reading time file after reading the CHANGELOG with date > /root/scripts/.last_changelog_read

Updating Shutdown Service

You might need to run systemctl deamon-reload after updating this repository. This will be noted in the CHANGELOG for your convenience.

Usage

The check and monitoring scripts in this repo can be run periodically be run and if any problems are detected, they produce output.

The output of these scripts can be redirected and used however you like. Typically I redirect the output to the helpers/tg_notify.sh script which notifies me of any noisy scripts. In case of expected repeating failures, I first redirect the ouput to helpers/state_machine.sh "keyword" which silences repeated messages. The "state machine" only saves a copy of the last message per keyword and compares it to the next message.

Regardless of any problems each script also logs its executions under logs/ Make sure you created this folder during installation.

Crontab Scheduling

You can schedule regular execution of these scripts with "cron". To configure cron use the crontab -e command, which opens VIM, our favourite text editor, containing the cron schedule. If you are unsure about the cron schedule, use Crontab Guru.

My current crontab looks like this:

* * * * * bash -c 'cd /root/admin-scripts && ./file_monitor.sh | ./helpers/tg_notify.sh'
*/2 * * * * bash -c 'cd /root/admin-scripts && ./monitoring.sh | ./helpers/state_machine.sh "monitoring" | ./helpers/tg_notify.sh'
*/4 * * * * bash -c 'cd /root/admin-scripts && ./dyndns.sh | ./helpers/tg_notify.sh'
*/3 * * * * bash -c 'cd /root/admin-scripts && ./system_health_check.sh | ./helpers/state_machine.sh "system" | ./helpers/tg_notify.sh'
15 * * * * bash -c 'cd /root/admin-scripts && ./docker_health_check.sh | ./helpers/state_machine.sh "docker" | ./helpers/tg_notify.sh'
*/15 * * * * bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'

@reboot sleep 10 && /root/admin-scripts/helpers/tg_notify.sh '[STARTUP] System just booted'
@reboot sleep 30 && bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh  | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'

Adapt this to your needs, you might also implement other checks and only use the helpers/tg_notify.sh script from this repo. Or you might implement your own notification script to notify you via another service. The helpers/tg_notify.sh can easily be adapted (just remove comment) to forward all notifications to STDOUT which typically makes cron send a mail.

Using Startup Helper

The post_startup.sh script is a helper script which should be run as soon as possible after a system boot. It will check for encrypted datasets with user provided passwords which are not mounted yet. After decrypting the datasets this script starts the configured docker containers which are depending on the encrypted datasets. Because my containers struggle with DNS which depends on a docker container itself, the script also restarts some configured docker containers at this point.

To complete the post startup tasks use the following command after logging in as your regular user:

./post_startup.sh

You can permit your user to run all of the commands in the script with out having to enter the users password by adapting the "sudoers" file. Beware that this is a security concern, as the regular user can thus manage all docker containers, which can hijack your host network, amongst other things. But as I connect to my servers via SSH keys which are stored reasonably securely on Yubikeys, I waive this security measure. To edit the sudoers file use the visudo command as root, which also opens VIM, our favourite text editor. My relevant part of the sudoers file looks as follows:

radioelephant ALL=NOPASSWD: /usr/bin/zfs mount -a -l
radioelephant ALL=(docker) NOPASSWD: /usr/bin/docker

Backups

Backup Scipts are not documented at this moment, see source code for more information. Expect changes as this is work in progress.

Contributors

• Robin Meier (robin@meier.si)