|Debian / ArchLinux Admin Helpers|
For easier management of Chaosdorf hosts.
The admin toolkit is a git tracked Debian package, which is installed on any host administrated by us (like the chaosdorf.de servers and the space's router). It provides the hosts with most of the required scripts and configs, which saves a lot of fiddling around in /usr/local and especially ensures that all hosts are up-to-date.
We use fabric to roll out package updates. The software still got some problems, but "fab configs deploy:2011.09.07" is way more comfortable than running scp and ssh for every host.
Our nagios checks run locally via cron, the results are transmitted via "send_nsca". They are configured for all hosts in one central file, which is parsed by a custom perl script.
Custom or not shipped by Debian.
- check_cert_expire reminds us when it's time to renew our SSL certificate. Much better than calendar entries.
- check_git_status makes sure noone forgot to commit their changes in /etc (etckeeper) or /usr/local (plain git repo). The check intervals are pretty long, so it doesn't interfere with administrative work.
- check_websites reads a little config to see if all URL / VHost combinations work or redirect as expected.
We have a bot called icinga in #chaosdorf. It's a simple perl script connected to Icinga, which immediately spams service alerts / recoveries into the channel. Handy if you check IRC more frequently than mail and if people wonder if something is b0rked or not.
After every system upgrade, a dpkg hook runs checkrestart to find daemons running with outdated library versions.
backup_external locally collects, compresses and encrypts important data and then transmits it to a remote host via SFTP. The backups are encrypted with a dedicated Admin key.
Like vim, etckeeper, git, zsh. So we don't need to install our standard software by hand (or maybe forget a package).