admin-scripts/README.md

Admin Scripts

This repo contains some administration scripts for administrering a debian machine. The covered tasks range from file change tracking via http/ssh monitoring to zfs health checking.

Installation

The scripts in this repo must be checked out into /root/scripts.

cd /root
git clone ssh://git@git.mneun.ch:2222/radioelephant/admin-scripts.git scripts
mkdir -p /root/logs

Config Files

For each script there is a .script_name_env.EXAMPLE file, which you must copy (remove .EXAMPLE part) and edit while providing your own information.

If you want to use the example configuration, you can symbolic link the files. This really only makes sense for .system_health_check. The command for this is:

cd /root/scripts
ln -s .system_health_check.EXAMPLE .system_health_check

Shutdown Notification

To be notified immediately before sever shutdown the contained run-before-shutdown.service can be installed as follows. This service file will send a telegram notification informing about the pending shutdown.

ln -s /root/scripts/run-before-shutdown.service /etc/systemd/system/shutdown.target.wants/run-before-shutdown.service
systemctl daemon-reload

Startup Helper

To install the startup helper script into the regular user home use the following commands, while supplying your username on the first line.

USRNAME=radioelephant
ln -s /root/scripts/post_startup.sh /home/$USRNAME/post_startup
cp /root/scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env
vim /home/$USRNAME/.post_startup_env

Just shutdown your computer if you don't know how to exit VIM at this point.

Updating

You can update the admin scripts by pulling from the git remote.

cd /root/scripts
git pull

Usage

The check and monitoring scripts in this repo can be run periodically be run and if any problems are detected, they produce output.

The output of these scripts can be redirected and used however you like. Typically I redirect the output to the telegram_notification.sh script which notifies me of any noisy scripts.

Regardless of any problems each script also logs its executions in /root/logs. Make sure you created this folder during installation.

Crontab

You can schedule regular execution of these scripts with "cron". To configure "cron" use the crontab -e command, which opens VIM, our favourite text editor, containing the cron schedule. If you are unsure about the cron schedule, use Crontab Guru.

My current crontab looks like this:

* * * * * bash -c '/root/scripts/file_monitor.sh | /root/scripts/telegram_notification.sh'
*/2 * * * * bash -c '/root/scripts/monitoring.sh | /root/scripts/telegram_notification.sh'
*/4 * * * * bash -c '/root/scripts/dyndns.sh | /root/scripts/telegram_notification.sh'
*/3 * * * * bash -c '/root/scripts/system_health_check.sh | /root/scripts/telegram_notification.sh'
15 * * * * bash -c '/root/scripts/docker_health_check.sh | /root/scripts/telegram_notification.sh'
*/15 * * * * bash -c '/root/scripts/zfs_health_check.sh | /root/scripts/telegram_notification.sh'

@reboot sleep 10 && /root/scripts/telegram_notification.sh '[STARTUP] System just booted'
@reboot sleep 30 && bash -c '/root/scripts/zfs_health_check.sh | /root/scripts/telegram_notification.sh'

Adapt this to your needs, you might also implement other checks and only use the telegram_notification.sh script from this repo. Or you might implement your own notification script to notify you via another service. The telegram_notification.sh can easily be adapted to forward all notifications to STDOUT which typically sends an email.

Startup

The post_startup.sh script is a helper script which should be run as soon as possible after a system boot. It will check for encrypted datasets with user provided passwords which are not mounted yet. After decrypting the datasets this script starts the configured docker containers which are depending on the encrypted datasets. Because my containers struggle with DNS which depends on a docker container itself, the script also restarts some configured docker containers at this point.

To complete the post startup tasks use the following command after logging in as your regular user:

./post_startup.sh

You can permit your user to run all of the commands in the script with out having to enter the users password by adapting the "sudoers" file. Beware that this is a security concern, as the regular user can thus manage all docker containers, which can hijack your host network, amongst other things. But as I connect to my servers via SSH keys which are stored reasonably securely on Yubikeys, I waive this security measure. To edit the "sudoers" file use the visudo command as root, which also opens VIM, our favourite text editor. My relevant part of the "sudoers" file looks as follows:

radioelephant ALL=NOPASSWD: /usr/bin/zfs mount -a -l
radioelephant ALL=(docker) NOPASSWD: /usr/bin/docker

Contributors

• Robin Meier (robin@meier.si)