admin-scripts/README.md

# Admin Scripts

This repo contains some administration scripts for administrering a debian machine.
The covered tasks range from file change tracking via http/ssh monitoring to zfs health checking.

*TODO: Add better description about host system and notification method in first sentence.*


## Installation

The scripts in this repo ***must*** be checked out into `/root/scripts`.

```bash
cd /root
git clone ssh://git@git.mneun.ch:2222/radioelephant/admin-scripts.git scripts
mkdir -p /root/logs
```

If you are unable to clone via SSH, try via https.
SSH is preferrable.

```bash
cd /root
git clone https://git.mneun.ch/radioelephant/admin-scripts.git scripts
mkdir -p /root/logs
```

Please also quickly read the [CHANGELOG](CHANGELOG.md) and confirm this by running `date > /root/scripts/.last_changelog_read`.
This will be helpful when updating the admin scripts later on.

### Config Files

For each script there is a `.[script_name]_env.EXAMPLE` file, which you must copy (remove `.EXAMPLE` part) and edit while providing your own information.

```bash
SCRIPT_NAME=zfs_health_check
cp /root/scripts/.${SCRIPT_NAME}_env.EXAMPLE /root/scripts/.${SCRIPT_NAME}_env
vim /root/scripts/.${SCRIPT_NAME}_env
```

If you want to use the example configuration, you could symbolic link the files instead of just copying them.
This really only makes sense for `.system_health_check`.
The command for this is:

```bash
cd /root/scripts
ln -s .system_health_check.EXAMPLE .system_health_check
```

### Shutdown Notification

To be notified immediately before sever shutdown the contained `run-before-shutdown.service` can be installed as follows.
This service file will send a telegram notification informing about the pending shutdown.

```bash
ln -s /root/scripts/run-before-shutdown.service /etc/systemd/system/shutdown.target.wants/run-before-shutdown.service
systemctl daemon-reload
```

### Startup Helper

To install the [startup helper script](#using-startup-helper) into the regular user home use the following commands, while supplying your username on the first line.

```bash
USRNAME=radioelephant
cp /root/scripts/post_startup.sh /home/$USRNAME/post_startup.sh
cp /root/scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/.post_startup_env
vim /home/$USRNAME/.post_startup_env
```

*Just shutdown your computer if you don't know how to exit VIM at this point.*

See also [Updating Startup Helper](#updating-startup-helper).

### Updating

You can update the admin scripts by pulling from the git remote.

```bash
cd /root/scripts
git pull
```

For most of the scripts you only need to check if the `.[script_name]_env.EXAMPLE` has changed and contains different keys than your copied `.[script_name]_env` file.
For your convenience, changes to environment variable files will be documented in the [CHANGELOG](CHANGELOG.md).
If you followed the instructions in this README, then you will find the last time you updated this repository in the `.last_changelog_read` file.
Read it with `cat /root/scripts/.last_changelog_read`.

**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**

#### Updating Startup Helper

The [startup helper script](#using-startup-helper) in the regular user home cannot be updated via `git pull` and needs to be compared and/or copied manually.
To compare use the following commands:

```bash
USRNAME=radioelephant
diff /root/scripts/post_startup.sh /home/$USRNAME/post_startup.sh
diff /root/scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env
```

To copy the [startup helper script](#using-startup-helper) again, use the following commands (first line not needed, if you just ran the last set of commands).
Please adapt the environment file `/home/[USRNAME]/.post_startup_env` manually.

```bash
USRNAME=radioelephant
cp /root/scripts/post_startup.sh /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
```

**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**

#### Updating Shutdown Service

You might need to run `systemctl deamon-reload` after updating this repository.
This will be noted in the CHANGELOG for your convenience.


## Usage

The check and monitoring scripts in this repo can be run periodically be run and if any problems are detected, they produce output.

The output of these scripts can be redirected and used however you like.
Typically I redirect the output to the `telegram_notification.sh` script which notifies me of any noisy scripts.

Regardless of any problems each script also logs its executions in `/root/logs`.
Make sure you created this folder during [installation](#installation).

### Crontab Scheduling

You can schedule regular execution of these scripts with "cron".
To configure cron use the `crontab -e` command, which opens VIM, *our favourite text editor*, containing the cron schedule.
If you are unsure about the cron schedule, use [Crontab Guru](https://crontab.guru).

My current crontab looks like this:

```crontab
* * * * * bash -c '/root/scripts/file_monitor.sh | /root/scripts/telegram_notification.sh'
*/2 * * * * bash -c '/root/scripts/monitoring.sh | /root/scripts/telegram_notification.sh'
*/4 * * * * bash -c '/root/scripts/dyndns.sh | /root/scripts/telegram_notification.sh'
*/3 * * * * bash -c '/root/scripts/system_health_check.sh | /root/scripts/telegram_notification.sh'
15 * * * * bash -c '/root/scripts/docker_health_check.sh | /root/scripts/telegram_notification.sh'
*/15 * * * * bash -c '/root/scripts/zfs_health_check.sh | /root/scripts/telegram_notification.sh'

@reboot sleep 10 && /root/scripts/telegram_notification.sh '[STARTUP] System just booted'
@reboot sleep 30 && bash -c '/root/scripts/zfs_health_check.sh | /root/scripts/telegram_notification.sh'
```

Adapt this to your needs, you might also implement other checks and only use the `telegram_notification.sh` script from this repo.
Or you might implement your own notification script to notify you via another service.
The `telegram_notification.sh` can easily be adapted (just remove comment) to forward all notifications to `STDOUT` which typically makes cron send a mail.

### Using Startup Helper

The `post_startup.sh` script is a helper script which should be run as soon as possible after a system boot.
It will check for encrypted datasets with user provided passwords which are not mounted yet.
After decrypting the datasets this script starts the configured docker containers which are depending on the encrypted datasets.
Because my containers struggle with DNS which depends on a docker container itself, the script also restarts some configured docker containers at this point.

To complete the post startup tasks use the following command after logging in as your regular user:

```bash
./post_startup.sh
```

You can permit your user to run all of the commands in the script with out having to enter the users password by adapting the "sudoers" file.
Beware that this is a security concern, as the regular user can thus manage all docker containers, which can hijack your host network, amongst other things.
But as I connect to my servers via SSH keys which are stored reasonably securely on Yubikeys, I waive this security measure.
To edit the sudoers file use the `visudo` command as root, which also opens VIM, *our favourite text editor*.
My relevant part of the sudoers file looks as follows:

```sudoers
radioelephant ALL=NOPASSWD: /usr/bin/zfs mount -a -l
radioelephant ALL=(docker) NOPASSWD: /usr/bin/docker
```

## Contributors

- Robin Meier (robin@meier.si)