TL;DR Using freeBSD is like sailing the seas, whereas linux is more like having fun with a brand new shiny boeing 737 MAX.So, what I like are factual, damn practical stories.
It began with a need : a damn bloody simple backup server.
And no complexities here : I have only linux and BSD computers that all have deployed ssh keys, hence rsync over ssh seems to be the « let's do first, improve later » kind of thinking.
It only requires ssh + rsync, git and a ... DNS resolver (unbound), because ISP DNS blocking is a thing in my country and a backup server might have to talk (git) to push/pull external data.
Unbound is cheap in resources, easy to install and a perfect use case for having fun with zfs thin jails.
Last time I had self hosted domain on a freebsd server I used ezjail because as a noob I flet secured.
Workinig with the freebsd handbook and the raw command is way more securing. I even did some nullfs mounting for the fun and for practicality. It is a sane alternative to symlinks.
Il also love to write the jails config file because their grammar is human readable and concise. It's almost like writing an haiku or a sonnet : a concise form of poetry.
Let's say that BSD being BSD, unices being unices, linux and bsd being close setting up the backup went fast. It was the work of a very-slow day from install to setup.
I did reinstall a few time, and I think interesting to share my though on disk encryption : why I wiped out my fully encrypted disk setup. It was impractical to cut myself from a reboot without a human behind the keyboard when a BACKUP server, might comme to be fucking necessary to be accessible. (I have lived abroad as an immigrant scared of being expulsed because of papers and I have a paranoïa about having backups of IDs first existing, and secondly to be easily accessible from the outside in case I am restricted the physical access to my house and need to prove my official identity FAST.).
So, yes, our use case of backup varies with our past experiences, including the ones of disasters that makes every sysadmins' risk sensitivity vary with their life.
So yes, you want to able to maintain a server while not having physical access to it, doing updates (of kernel drivers for instance) that may require reboot. And, you can still cipher the home dirs of the users.
So practicallity beats purity : physical access to my computer is easy and has the boot invite in single mode for password recovery, enabling root to impersonate and read the data. On the other hand, my physical threat model inherited from my life trauma is not someone breaking into my house (I have experienced it), but me being taken out of my house and needing urgently papers to prove my identity.
Plus in case of burglary, my server is a broken 15 years old computer, with a broken screen and a centrino sticker. Not really the kind of stuff you steal.
So yeah, I have a huge security hole when it comes to the physical dimension based on my risk assumptions which diverge from common experience.
As you can see, from the week end to monday, I filled 60% of a disk with backup, and I also installed a supervision, and a web server, even though they are increasing uselessly the surface of vulnerability.
It's one of my divergence with a lot of sysadmins that love to spin kubernetes or heavy containers with one VM per task. Me, I prefer my jails : they are both a container for security, but also for the mess of depencies.
You see thanks to zfs, each jails cost me less in terms of disk space, they are almost free after the first one, the same cannot be said of docker, qemu and vmware. My jails balloon (take more shared space) on the disk without me wasting space, and energy supervising hosts and guests' disks space just « in case ».
Also, jails are a localized mess due to the depency hell. They sometimes require modifications in rc.conf that are not wishable on the host or depencies to like wayland that you would like to disappear if you change your mind about this package. Jails are not only software containers, they are also a container of the mess of installing to much service on an operating system. And with /usr/ports I can have a nice and boring safe and conservative behaviour with the host sysadmin management and get totally nuts with bleeding edge versions in some jails without messing up with the host.
And since it's only chroot, I can copy files from all jails and the hosts without any layer of complexity.
So at 9am the backups kick their life, because, by reminiscence of my past life as a poor, I did a 9 to 5 server. The kind that can be shut down to let people sleep, or to be moved away fast without anything breaking.
The kind of companion that silently slurps data being pushed to it if it is there. The advantage of rsync over ssh from the backuped computer is that on the backuped computers you have a live memento in the crontab of your prefered arguments on rsync to deploy a new backup server.
Most of my difficulties in my plan has been the lack of cooperation of linux in overriding the dhcp settings to supersede the local dns. Modern Linux are pissing me off when it comes to DNS configuration. At least with BSD I know /etc/resolv.conf is the solution. Because, SINCE I am sysadmin, I love making X509 certificates that are self signed and REQUIRE a DNS warrantying that the direct name and RPTR for the IP are same.
I don't like HTTPS, but you have to fucking idiot to let a plain http server in a setup where you put HUGE security holes. And you want your certificate to be included in the openssl wallet, as well as firefox. So you do it slow and nice..
# Mordor authority on .home
openssl req -x509 -nodes -new -sha256 -days 1024 -newkey rsa:2048 -keyout RootCA.key -out RootCA.pem -subj "/C=VX/CN=Mordor"
# in em format
openssl x509 -outform pem -in RootCA.pem -out RootCA.crt
touch home.ext
# make me a certificate for www.home
openssl req -new -nodes -newkey rsa:2048 -keyout www.home.key -out www.home.csr -subj "/C=VX/ST=Vexin/L=Toulouse/O=Sauron/CN=www.home"
openssl x509 -req -sha256 -days 1024 -in www.home.csr -CA RootCA.pem -CAkey RootCA.key -CAcreateserial -extfile home.ext -out www.home.crt
# put in the jail
cp www.home.{crt,key} /usr/local/jails/containers/www/usr/local/etc/nginx/
# restart the nginx in the jails www
jexec www service nginx restart
# update this dir that is used by openssl to store openssl certificate
openssl rehash .
I know solutions like lemur exists to handle SSL Authority and that they are awesome. But the lazier I am with openssl command line, the less proficient I am when all hell break lose and I need to fire muscle memory with openssl to manipulate or check certificate. So now it's monday, 10 am, I am doing my ssl certificate for my nginx because I decided to install munin.
Munin is a very old suboptimal supervising solution that has not been updated in years, I use it as a first tool because :
- I know basic probes cover 95% of my needs
- I know how to write a plugin fast for it
But I always rush to munin because I know I can deploy it fast and that it generates STATIC web pages with a crontab (meaning I don't need any cgi's PHP or java). And it can gather up data out of the box from the other stations (thanks to broadcast).
Well, a linux sysadmin, I was disappointed munin on freeBSD does not covers as much. I had my most important ones :
- obvious ressources (RAM/CPU/Disks/Net)
- SMARTD - you should NEVER underestimate how much SMART (self reporting tools for hard drive) are important for non SSD hard drive.
- ACPI temperature
- fan speed
Raised as a physicists I don't do any graphs without label. So I asked miod (the most reverred guru of french BSD community) where the data were.
And the guy said
Selon acpi_ibm(4) :
dev.acpi_ibm.0.thermal
(read-only) Shows the readings of up to eight different
temperature sensors. Most ThinkPads include six or more
temperature sensors but only expose the CPU temperature through
acpi_thermal(4). Some ThinkPads have the below sensor layout
which might vary depending on the specific model:
1. CPU
2. Mini PCI Module
3. HDD
4. GPU
5. Built-in battery
6. UltraBay battery
7. Built-in battery
8. UltraBay battery
And that's the nicest RTFM I have experienced in my life. And that should remind you of one of the FUNDAMENTAL difference between linux and BSD.
The fresh up to date information is almost always in your man page and not on the internet.
Which lead me to thank him, and move my lazy ass to write the plugin out using the linux ACPI as a template in 30 minutes. I love the BSD community they are always nice.
Here is the code.
Here is the graph :
And here, just after the munin plugin kicks off : you see me panicked at the view of the CPU burning at 80°C and put a fan on the computer to make it cool down at more reasonable temperature.
I very swiftly reacted and checked if by any chance I did not missed a package to activate the fan. Of course I missed the bsdfan package, that lacks a service so you have to write a service.
It took me 5 minutes of googling to have the script :
[root@petiot /etc/ssl/home]#
more /etc/rc.d/bsdfan
#!/bin/sh
#
#
# PROVIDE: bsdfan
# REQUIRE: DAEMON
# BEFORE: LOGIN
# KEYWORD: nojail shutdown
. /etc/rc.subr
name="bsdfan"
desc="Modify the fan according to temperature"
rcvar="bsdfan_enable"
command="/usr/local/bin/${name}"
bsdfan_flags="-d"
load_rc_config $name
run_rc_command "$1"
It damn beats the speed at which I can deploy a random script as a service with systemd. And by removing the external fan and experiencing the drop of termerature I had a clear hint it was working.So for fun I added a probe on the rotation of the fans :
Well, at the end of the day, this caveat made me like freeBSD even more.
Having to improvise adapt, overcome it part of the sysadmin life. Having a small turbulence at the beginning of a life of a server is a good way of battle proving the choice.
At the opposite of linux, BSD relies a lot on shell (/bin/sh) as a DSL for the config and setup as a one ring to rule them all.
On the other hand CLI experienc of linux is way more diverse having json/ini files/custom XML ... as a text interface to interact with the core of the system. The BSD clearly is more consistent and easy to use from command line making it easy for you to customize it in a clean way. You have the impression to wake up from an heavily loaded smoke of bullshit when you realize how simple containerisation can be, tweaking the system can be, in a sane way.
My biggest grief with linux right now is not technical, but the user experience on the command line that is an absolute mess. I hate to have more than one way to setup network bridges (ip2+brctl) I totally love to do it with ifconfig in a consistent way.
I have to learn one grammar for parsing and templating most of the sysadmin tools/config : and it's almost always /bin/sh feeling. It really remove the accidental brain fog that stands between me and getting things done in a short time without a mistake.
Chroots may not be as sexy as docker, but they are lighter for an almost equivalent service and awesomely well integrated with the os, you should use them more. I can even deploy debian based jails finger in ze nose with linux jails, while linux sysadmins require hypervirtualization (qemu, vmware) to deploy a freebsd box among their servers.
In terms of sysadmin experience, freeBSD is like removing a lot of brain fog coming from over-complication, it is simpler, more conservative and easier to hack CLEANLY.
I really encourage you to have fun with my automated freebsd customisation for qemu.
PS thanks to benoit for this server.
PPS : I feel so relieved to have my backups again.
