# Admin Scripts

This repo contains some administration scripts for administrering a debian machine.
The covered tasks range from file change tracking via http/ssh monitoring to zfs health checking.

*TODO: Add better description about host system and notification method in first sentence.*


## Installation

This repository can be checked out wherever you like.
Typically this is `~/admin-scripts`.
The rest of this README assumes you checked out this repo into `/root/admin-scripts`.

```bash
cd
git clone ssh://git@git.mneun.ch:2222/radioelephant/admin-scripts.git
```

If you are unable to clone via SSH, try via https.
SSH is preferrable.

```bash
cd
git clone https://git.mneun.ch/radioelephant/admin-scripts.git
```

You might need to install some utilities to be able to use the scripts in this repository.
To install all required dependencies, use the following command:

```bash
sudo apt install moreutils bc lm-sensors
sudo sensors-detect --auto
```

To make sure a script is runnable, always run it manually first, before setting up cronjobs.
If the script produces any unexpected output, you might be missing utilities or the configuration.

Please also quickly read the [CHANGELOG](CHANGELOG.md) and confirm this by running `date > /root/scripts/.last_changelog_read`.
This will be helpful when updating the admin scripts later on.

### Config Files

Config files are located in the `config/` directory.
Each script has its own cofiguration file, in this repo there only are the `config/[script_name].EXAMPLE` example configuration files.
For each script you want to use, you must copy the example and fill in your own data.
For example (`monitoring.sh`):

```bash
cd ~/admin-scripts
cp config/monitoring.EXAMPLE config/monitoring
vim config/monitoring
```

If you want to use the provided example configuration, you could symbolic link the files instead of just copying them.
This really only makes sense for `system_health_check` and `state_machine`.
The command for this is:

```bash
cd ~/admin-scripts
ln -s config/system_health_check.EXAMPLE config/system_health_check
```

### Shutdown Notification

To be notified immediately before sever shutdown the contained `run-before-shutdown.service` can be installed as follows.
This service file will send a telegram notification informing about the pending shutdown.

```bash
ln -s /root/admin-scripts/run-before-shutdown.service /etc/systemd/system/shutdown.target.wants/run-before-shutdown.service
systemctl daemon-reload
```
*Adapt the above command, if your scripts are located somewhere else!*

### Startup Helper

To install the [startup helper script](#using-startup-helper) into the regular user home use the following commands, while supplying your username on the first line.

```bash
USRNAME=radioelephant
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
cp /root/admin-scripts/config/post_startup.EXAMPLE /home/$USRNAME/.post_startup_env
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/.post_startup_env
vim /home/$USRNAME/.post_startup_env
```

*Just shutdown your computer if you don't know how to exit VIM at this point.*

See also [Updating Startup Helper](#updating-startup-helper).

### Updating

You can update the admin scripts by pulling from the git remote.

```bash
cd ~/admin-scripts
git pull
```

For most of the scripts you only need to check if the example configuration file (i.e. `config/monitoring.EXAMPLE`) has changed and contains different keys than your copy (i.e. `config/monitoring`).
For your convenience, changes to configuration files will be documented in the [CHANGELOG](CHANGELOG.md).
If you followed the instructions in this README, then you will find the last time you pulled this repository and read the [CHANGELOG](CHANGELOG.md) in the `.last_changelog_read` file.
Get the value with `cat /root/admin-scripts/.last_changelog_read`.

**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**

#### Updating Startup Helper

The [startup helper script](#using-startup-helper) in the regular user home cannot be updated via `git pull` and needs to be compared and/or copied manually.
To compare use the following commands:

```bash
USRNAME=radioelephant
diff /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
diff /root/admin-scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env
```

To copy the [startup helper script](#using-startup-helper) again, use the following commands (first line not needed, if you just ran the last set of commands).
Please adapt the environment file `/home/[USRNAME]/.post_startup_env` manually.

```bash
USRNAME=radioelephant
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
```

**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**

#### Updating Shutdown Service

You might need to run `systemctl deamon-reload` after updating this repository.
This will be noted in the CHANGELOG for your convenience.


## Usage

The check and monitoring scripts in this repo can be run periodically be run and if any problems are detected, they produce output.

The output of these scripts can be redirected and used however you like.
Typically I redirect the output to the `helpers/tg_notify.sh` script which notifies me of any noisy scripts.
In case of expected repeating failures, I first redirect the ouput to `helpers/state_machine.sh "keyword"` which silences repeated messages.
The "state machine" only saves a copy of the last message per keyword and compares it to the next message.

Regardless of any problems each script also logs its executions under `logs/`
Make sure you created this folder during [installation](#installation).

### Crontab Scheduling

You can schedule regular execution of these scripts with "cron".
To configure cron use the `crontab -e` command, which opens VIM, *our favourite text editor*, containing the cron schedule.
If you are unsure about the cron schedule, use [Crontab Guru](https://crontab.guru).

My current crontab looks like this:

```crontab
* * * * * bash -c 'cd /root/admin-scripts && ./file_monitor.sh | ./helpers/tg_notify.sh'
*/2 * * * * bash -c 'cd /root/admin-scripts && ./monitoring.sh | ./helpers/state_machine.sh "monitoring" | ./helpers/tg_notify.sh'
*/4 * * * * bash -c 'cd /root/admin-scripts && ./dyndns.sh | ./helpers/tg_notify.sh'
*/3 * * * * bash -c 'cd /root/admin-scripts && ./system_health_check.sh | ./helpers/state_machine.sh "system" | ./helpers/tg_notify.sh'
15 * * * * bash -c 'cd /root/admin-scripts && ./docker_health_check.sh | ./helpers/state_machine.sh "docker" | ./helpers/tg_notify.sh'
*/15 * * * * bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'

@reboot sleep 10 && /root/admin-scripts/helpers/tg_notify.sh '[STARTUP] System just booted'
@reboot sleep 30 && bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh  | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'
```

Adapt this to your needs, you might also implement other checks and only use the `helpers/tg_notify.sh` script from this repo.
Or you might implement your own notification script to notify you via another service.
The `helpers/tg_notify.sh` can easily be adapted (just remove comment) to forward all notifications to `STDOUT` which typically makes cron send a mail.

### Using Startup Helper

The `post_startup.sh` script is a helper script which should be run as soon as possible after a system boot.
It will check for encrypted datasets with user provided passwords which are not mounted yet.
After decrypting the datasets this script starts the configured docker containers which are depending on the encrypted datasets.
Because my containers struggle with DNS which depends on a docker container itself, the script also restarts some configured docker containers at this point.

To complete the post startup tasks use the following command after logging in as your regular user:

```bash
./post_startup.sh
```

You can permit your user to run all of the commands in the script with out having to enter the users password by adapting the "sudoers" file.
Beware that this is a security concern, as the regular user can thus manage all docker containers, which can hijack your host network, amongst other things.
But as I connect to my servers via SSH keys which are stored reasonably securely on Yubikeys, I waive this security measure.
To edit the sudoers file use the `visudo` command as root, which also opens VIM, *our favourite text editor*.
My relevant part of the sudoers file looks as follows:

```sudoers
radioelephant ALL=NOPASSWD: /usr/bin/zfs mount -a -l
radioelephant ALL=(docker) NOPASSWD: /usr/bin/docker
```

### Backups

*Backup Scipts are not documented at this moment, see source code for more information. Expect changes as this is work in progress.*

## Contributors

- Robin Meier (robin@meier.si)