203 lines
8.9 KiB
Markdown
203 lines
8.9 KiB
Markdown
# Admin Scripts
|
|
|
|
This repo contains some administration scripts for administrering a debian machine.
|
|
The covered tasks range from file change tracking via http/ssh monitoring to zfs health checking.
|
|
|
|
*TODO: Add better description about host system and notification method in first sentence.*
|
|
|
|
|
|
## Installation
|
|
|
|
This repository can be checked out wherever you like.
|
|
Typically this is `~/admin-scripts`.
|
|
The rest of this README assumes you checked out this repo into `/root/admin-scripts`.
|
|
|
|
```bash
|
|
cd
|
|
git clone ssh://git@git.mneun.ch:2222/radioelephant/admin-scripts.git
|
|
```
|
|
|
|
If you are unable to clone via SSH, try via https.
|
|
SSH is preferrable.
|
|
|
|
```bash
|
|
cd
|
|
git clone https://git.mneun.ch/radioelephant/admin-scripts.git
|
|
```
|
|
|
|
You might need to install some utilities to be able to use the scripts in this repository.
|
|
To install all required dependencies, use the following command:
|
|
|
|
```bash
|
|
sudo apt install moreutils bc lm-sensors
|
|
sudo sensors-detect --auto
|
|
```
|
|
|
|
To make sure a script is runnable, always run it manually first, before setting up cronjobs.
|
|
If the script produces any unexpected output, you might be missing utilities or the configuration.
|
|
|
|
Please also quickly read the [CHANGELOG](CHANGELOG.md) and confirm this by running `date > /root/scripts/.last_changelog_read`.
|
|
This will be helpful when updating the admin scripts later on.
|
|
|
|
### Config Files
|
|
|
|
Config files are located in the `config/` directory.
|
|
Each script has its own cofiguration file, in this repo there only are the `config/[script_name].EXAMPLE` example configuration files.
|
|
For each script you want to use, you must copy the example and fill in your own data.
|
|
For example (`monitoring.sh`):
|
|
|
|
```bash
|
|
cd ~/admin-scripts
|
|
cp config/monitoring.EXAMPLE config/monitoring
|
|
vim config/monitoring
|
|
```
|
|
|
|
If you want to use the provided example configuration, you could symbolic link the files instead of just copying them.
|
|
This really only makes sense for `system_health_check` and `state_machine`.
|
|
The command for this is:
|
|
|
|
```bash
|
|
cd ~/admin-scripts
|
|
ln -s config/system_health_check.EXAMPLE config/system_health_check
|
|
```
|
|
|
|
### Shutdown Notification
|
|
|
|
To be notified immediately before sever shutdown the contained `run-before-shutdown.service` can be installed as follows.
|
|
This service file will send a telegram notification informing about the pending shutdown.
|
|
|
|
```bash
|
|
ln -s /root/admin-scripts/run-before-shutdown.service /etc/systemd/system/shutdown.target.wants/run-before-shutdown.service
|
|
systemctl daemon-reload
|
|
```
|
|
*Adapt the above command, if your scripts are located somewhere else!*
|
|
|
|
### Startup Helper
|
|
|
|
To install the [startup helper script](#using-startup-helper) into the regular user home use the following commands, while supplying your username on the first line.
|
|
|
|
```bash
|
|
USRNAME=radioelephant
|
|
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
|
|
cp /root/admin-scripts/config/post_startup.EXAMPLE /home/$USRNAME/.post_startup_env
|
|
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
|
|
chown $USRNAME:$USRNAME /home/$USRNAME/.post_startup_env
|
|
vim /home/$USRNAME/.post_startup_env
|
|
```
|
|
|
|
*Just shutdown your computer if you don't know how to exit VIM at this point.*
|
|
|
|
See also [Updating Startup Helper](#updating-startup-helper).
|
|
|
|
### Updating
|
|
|
|
You can update the admin scripts by pulling from the git remote.
|
|
|
|
```bash
|
|
cd ~/admin-scripts
|
|
git pull
|
|
```
|
|
|
|
For most of the scripts you only need to check if the example configuration file (i.e. `config/monitoring.EXAMPLE`) has changed and contains different keys than your copy (i.e. `config/monitoring`).
|
|
For your convenience, changes to configuration files will be documented in the [CHANGELOG](CHANGELOG.md).
|
|
If you followed the instructions in this README, then you will find the last time you pulled this repository and read the [CHANGELOG](CHANGELOG.md) in the `.last_changelog_read` file.
|
|
Get the value with `cat /root/admin-scripts/.last_changelog_read`.
|
|
|
|
**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**
|
|
|
|
#### Updating Startup Helper
|
|
|
|
The [startup helper script](#using-startup-helper) in the regular user home cannot be updated via `git pull` and needs to be compared and/or copied manually.
|
|
To compare use the following commands:
|
|
|
|
```bash
|
|
USRNAME=radioelephant
|
|
diff /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
|
|
diff /root/admin-scripts/.post_startup_env.EXAMPLE /home/$USRNAME/.post_startup_env
|
|
```
|
|
|
|
To copy the [startup helper script](#using-startup-helper) again, use the following commands (first line not needed, if you just ran the last set of commands).
|
|
Please adapt the environment file `/home/[USRNAME]/.post_startup_env` manually.
|
|
|
|
```bash
|
|
USRNAME=radioelephant
|
|
cp /root/admin-scripts/post_startup.sh /home/$USRNAME/post_startup.sh
|
|
chown $USRNAME:$USRNAME /home/$USRNAME/post_startup.sh
|
|
```
|
|
|
|
**Make sure to update the last reading time file after reading the CHANGELOG with `date > /root/scripts/.last_changelog_read`**
|
|
|
|
#### Updating Shutdown Service
|
|
|
|
You might need to run `systemctl deamon-reload` after updating this repository.
|
|
This will be noted in the CHANGELOG for your convenience.
|
|
|
|
|
|
## Usage
|
|
|
|
The check and monitoring scripts in this repo can be run periodically be run and if any problems are detected, they produce output.
|
|
|
|
The output of these scripts can be redirected and used however you like.
|
|
Typically I redirect the output to the `helpers/tg_notify.sh` script which notifies me of any noisy scripts.
|
|
In case of expected repeating failures, I first redirect the ouput to `helpers/state_machine.sh "keyword"` which silences repeated messages.
|
|
The "state machine" only saves a copy of the last message per keyword and compares it to the next message.
|
|
|
|
Regardless of any problems each script also logs its executions under `logs/`
|
|
Make sure you created this folder during [installation](#installation).
|
|
|
|
### Crontab Scheduling
|
|
|
|
You can schedule regular execution of these scripts with "cron".
|
|
To configure cron use the `crontab -e` command, which opens VIM, *our favourite text editor*, containing the cron schedule.
|
|
If you are unsure about the cron schedule, use [Crontab Guru](https://crontab.guru).
|
|
|
|
My current crontab looks like this:
|
|
|
|
```crontab
|
|
* * * * * bash -c 'cd /root/admin-scripts && ./file_monitor.sh | ./helpers/tg_notify.sh'
|
|
*/2 * * * * bash -c 'cd /root/admin-scripts && ./monitoring.sh | ./helpers/state_machine.sh "monitoring" | ./helpers/tg_notify.sh'
|
|
*/4 * * * * bash -c 'cd /root/admin-scripts && ./dyndns.sh | ./helpers/tg_notify.sh'
|
|
*/3 * * * * bash -c 'cd /root/admin-scripts && ./system_health_check.sh | ./helpers/state_machine.sh "system" | ./helpers/tg_notify.sh'
|
|
15 * * * * bash -c 'cd /root/admin-scripts && ./docker_health_check.sh | ./helpers/state_machine.sh "docker" | ./helpers/tg_notify.sh'
|
|
*/15 * * * * bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'
|
|
|
|
@reboot sleep 10 && /root/admin-scripts/helpers/tg_notify.sh '[STARTUP] System just booted'
|
|
@reboot sleep 30 && bash -c 'cd /root/admin-scripts && ./zfs_health_check.sh | ./helpers/state_machine.sh "zfs" | ./helpers/tg_notify.sh'
|
|
```
|
|
|
|
Adapt this to your needs, you might also implement other checks and only use the `helpers/tg_notify.sh` script from this repo.
|
|
Or you might implement your own notification script to notify you via another service.
|
|
The `helpers/tg_notify.sh` can easily be adapted (just remove comment) to forward all notifications to `STDOUT` which typically makes cron send a mail.
|
|
|
|
### Using Startup Helper
|
|
|
|
The `post_startup.sh` script is a helper script which should be run as soon as possible after a system boot.
|
|
It will check for encrypted datasets with user provided passwords which are not mounted yet.
|
|
After decrypting the datasets this script starts the configured docker containers which are depending on the encrypted datasets.
|
|
Because my containers struggle with DNS which depends on a docker container itself, the script also restarts some configured docker containers at this point.
|
|
|
|
To complete the post startup tasks use the following command after logging in as your regular user:
|
|
|
|
```bash
|
|
./post_startup.sh
|
|
```
|
|
|
|
You can permit your user to run all of the commands in the script with out having to enter the users password by adapting the "sudoers" file.
|
|
Beware that this is a security concern, as the regular user can thus manage all docker containers, which can hijack your host network, amongst other things.
|
|
But as I connect to my servers via SSH keys which are stored reasonably securely on Yubikeys, I waive this security measure.
|
|
To edit the sudoers file use the `visudo` command as root, which also opens VIM, *our favourite text editor*.
|
|
My relevant part of the sudoers file looks as follows:
|
|
|
|
```sudoers
|
|
radioelephant ALL=NOPASSWD: /usr/bin/zfs mount -a -l
|
|
radioelephant ALL=(docker) NOPASSWD: /usr/bin/docker
|
|
```
|
|
|
|
### Backups
|
|
|
|
*Backup Scipts are not documented at this moment, see source code for more information. Expect changes as this is work in progress.*
|
|
|
|
## Contributors
|
|
|
|
- Robin Meier (robin@meier.si)
|