My objective is to put together a highly secure and usable server environment, that uses a SQL backend as much as possible for "everyday" routines, such as authentication, as well as storage for user account information. My aim was also to do all this with a very low budget, because I really didn't have a lot of money to spend. I'm well aware that you could do most of this by just using the ports collection, but sometimes compiling by hand is required, e.g. when you need to set a compile option not present in the ports version.
While there are some very good directory access protocols, such as LDAP, I've always preferred SQL databases, due to the fact that they are relatively widely supported and do an excellent job at storing and retrieving data.
The basic assumption here is that "everything is insecure by default".
If you're going to attempt a similar setup, I'll assume that you have knowledge of compiling software from source, installing FreeBSD, how jails work in FreeBSD and basic TCP/IP networking. This setup is also well suited for a VPS provider, such as DigitalOcean.
Throughout this document, you will see either hash symbols (#) or dollar signs ($) in front of the commands. These indicate whether the command should be run as a regular user or with superuser privileges (i.e. root). Do not include these in the commands! And by the way, copy-pasting commands is not such a great idea. I also provide my configuration files here for reference only. Please do not just copy them over to your server without fully understanding what they do. Additionally, please do not use the authors of this documents as a technical support, which we are most certainly not. Instead, you should consult the respective softwares' manual pages, documentations, forums and mailing lists.
Use the information in this document at your own risk. I disavow any potential liability for the contents of this document. Use of the concepts, examples, and/or other content of this document is entirely at your own risk.
All copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.
Naming of particular products or brands should not be seen as endorsements.
You are strongly recommended to make a backup of your system before major
installation and should make backups at regular intervals.
I used to prefer Linux as a server platform, that is, until the day I came in contact with FreeBSD. While Linux is not a bad alternative, some areas of it are still very under-developed. I once tried setting up quotas on an ext3 filesystem and a stable 2.4.x series kernel, only to find out that you would have needed plenty of kernel patches at the time. Not an option in my books.
Windows was never an option for me, due to the fact that it can be very inflexible on some areas. The stability I won't even bother to go in on here.
While Linux still remains my number one workstation operating system, I've come to think of FreeBSD as the ultimate low-budget server operating system. It offers rock-solid stability most of the time, has really nice security features such as jailcells and kernel security levels, and has a bunch of useful performance-enhanching tools, such as the Vinum volume manager, which implements software RAID.
I was also considering OpenBSD, which is said to be "secure by default". OpenBSD does however not include the 'jail' functionality (other than 'chroot', that is). OpenBSD does, however, include a lot of seurity patches, claming that a 'jail'-like functionality is not needed in this case. I feel, that the 'jail' functionality is not equal to security, but more of a fail-safe against configuration error or coding mistakes, which could quite possibly happen. Unfortunately, OpenBSD lacks functional PAM and nsswitch implementations, thus making it hard if not impossible to authenticate against a SQL database.
In my books, NetBSD falls between OpenBSD and FreeBSD, not really offering
This is most certainly not an easy choice to make, as a reliable database backend is the foundation for the server, and it's very difficult to change the backend when the system is running a production environment.
At the moment, MySQL seems to have the best support of the SQL databases. MySQL is also relatively easy to setup, but at the same time a bit more limited than, for example, PostgreSQL. In an ideal situation, the MySQL server should be replicated synchronously, but unfortunately - while such solutions do exist - that is a solution that comes at a quite heavy price tag.
PostgreSQL offeres open-source synchronous replication through their Postgres-R project.
However, there are several other aspects one must consider. One of these
is software support. One must remember, that while for example the Exim MTA
supports both MySQL and PostgreSQL, the Qpopper POP3 daemon can only be patched
for MySQL support. Of course, at this point, it might be good to check out the
competition, in order to see if someone else has implemented support for the
database backend you're going to use.
The basic services I wish to offer are the following:
FTP is mostly here for legacy reasons only. I've been meaning to replace it with SSH2's SFTP for a long time, but now that you can do TLS/SSL over FTP, the FTP service seems to die hard. In a high-security server, TLS/SSL is the ONLY way to properly do FTP, if it's non-anonymous.
These services will all need some sort of user accounts. Organising the accounts in a reasonable way is a challenging task of it's own. As my focus is on SQL database backend, I wanted to store all user accounts in a SQL server. I did some research on some PAM (Pluggable Authentication Module) solutions, but discovered, that they were too inflexible for a solution such as this, due to the fact that they only provide an authentication method, whereas I wanted to get rid of the whole flat-file /etc/users and /etc/passwd system. I stumbled upon NSS modules while searching for a solution, and found libnss-mysql, which is perfect for the task.
The problem with flat-file /etc/users and /etc/passwd is that it's difficult to replicate the user base to another server. This problem is easily circumvented when using SQL replication with NSS-level authentication and account management. However, this leads to another problem. What happens if the SQL server is unreachable? The whole account system will become unusable, effectively crippling the offered services; hence my need for a replicated SQL server environment.
I have divided the accounts the above services require into four basic account levels:
Jailed system accounts:
Jailed FTP accounts:
From the SQL point of view, the above setup will generate three tables; one for authenticating system (SSH/FTP) users, one for authenticating mail (SMTP/POP3/IMAP4) users, and finally one, which actually already exists in the default MySQL installation - the internal MySQL user database.
By placing /tmp and /var/log on their own partitions, we can easily avoid 'log-owerflow' attacks. In these types of attacks, the idea is to make attacked deamon fill up the disk partition with logs, simply by submitting requests to it in a rapid pace - for exmaple - each second, for an extended amount of time. At first I though about creating separate partitions for all jailed /tmp and /var/log directories, in the following fashion:
Mount point swap / /usr /tmp /var/jail/www_subsystem/tmp /var/jail/www_subsystem/var/log /var/jail/mail_subsystem/tmp /var/jail/mail_subsystem/var/log /var/jail/sql_subsystem/tmp /var/jail/sql_subsystem/var/log /var/jail/bnc_subsystem /var/log /var
However, the clear disadvantage of this approach is the scattered log files and - obviously - the need for multiple, relatively small partitions. Also, managing this many partitions using bsdlabel is cumbersome. After some researching, I found that the -l argument of syslogd would do the trick, thus allowing me to use the following partition layout:
Mount point swap / /usr /tmp /var/log /var
The root filesystem of the host system should be placed on it's own partiton, due to the fact that any media errors that may occur during disk I/O to user files will corrupt the filesystem containing vital system files as well. One might even go as far as to create separate root partitions for all jails as well, but I think that's overkill.
I have divided my server setup into several parts, which I will call subsystmes. Each subsystem is actually a FreeBSD jailcell, running as few services as possible. Ideally, the jail/service ratio would be 1:1, but this is not possible in practice (read below why). Creating the jailcells is quite a bit of work, but pays off in terms of added security. The basic subsystems I'll need for the above services are as follows:
This is by far the largest subsystem, due to the fact that many of the daemons need fileystem access to the same files. The daemons I need inside this jailcell are as follows:
The setup of this subsystem is documented in detail in my Mini How-To.
This is the second largest subsystem right after the webserver subsystem. The separation of the webserver files and maildirs allows me to specify separate quotas for both. The quota and account settings for mail are stored in a custom SQL table, which the following daemons will manipulate:
The setup of this subsystem is documented in detail in my
Exim, Amavisd-new, Courier-IMAP with TLS+MySQL Auth Mini How-To.
Amavisd-new is placed in it's own chroot on the host system (no need for a full jail), as communication with Exim is done via SMTP and communication with SQL via TCP. Amavisd-new has native SQL support though Perl's DBI. The anti-virus scanner(s) must also reside inside this chroot. I will use F-Prot as my anti-virus scanner, because it's simple, gets the job done and mostly because I happen to have a license for it.
The setup of this subsystem is discussed also in my .
This subsystem will contain the heart and brains of the server, namely the SQL server. Communication with the 'outside world' is done via TCP.
The one big problem with MySQL is, that it hasn't got quota support. However, since MySQL lives inside the UFS file system, you can implement quotas by chown'ing databases to appropriate users and activating user quotas for the disk partition which the MySQL databases live on. You don't need to worry about this leaving your database in an
inconsistent state when the quota is exceeded, as MySQL should then behave as in a "disk full" situation. The exact behaviour is documented in the MySQL manual.
Organizing the directory structure is the most difficult of all tasks. My vision was to have per-user quotas, not per-website quotas. This approach has the advantage of a user being able to control several websites using only one login. The disadvantage is that each DocumentRoot needs to be looked up somewhere, as we cannot produce a valid path dynamically, for exmaple, by using Apache's mod_rewrite's 'Mass Virtual Hosting'-technique.
At first I thought about solving the problem by using Apache's mod_rewrite to lookup paths using an External Rewriting Program, which actually was a Perl script, that pulls the data from my SQL table, which contains the DocumentRoots for all hosted virtual domains. That same SQL table is used by my AwStatsSQLBatch Perl script, which builds statistics for appropriate virtual domains.
Later, after some more thinking, I ditched that idea, and implemented mod_perl into my Apache httpd.conf file, thus allowing me to make the lookups straight from there.
Files created by content management systems (CMS) normally get the web server's UID/GID, and thus the files created by the CMS are not subject to quota. To solve this problem, PHP needs to be run using mod_suexec, which then runs the CMS with the user's UID/GID.
I decided to place all my jails (subsystems) under /var/jail/, and thus I ended up with a directory layout such as this.
User maildirs, access via POP3 and IMAP4 only, one login/mail address:
| `-- /var/jail/mail_subsystem/ | | var/mail/domain.tld/username/
User websites, access via SSH and FTP only:
| `-- /var/jail/www_subsystem | | /home/username | | /bin /usr/bin -> /bin /domain1.tld/public_html/ /domain2.tld/public_html/
The above directory structure is pretty much the only way to implement my vision of being able to administer several websites using one system (SSH/FTP) login. However, this layout leads to yet another problem. I wanted to chroot my FTP/SSH users to their own home directory inside the jailcell, for example /var/jail/www_subsystem/home/username. Thus, the users will only be able to see their own websites, and not other peoples. I do believe that chroot inside a jail is pure madness, but I couldn't think of any other way to implement this 'all-in-one' solution. Can you feel that paranoia running wild already? However, a problem arises when we want SSH functionality.
Providing a functional SSH shell is tricky when the users are chrooted into their own home directory, as they don't have access to /bin and /usr/bin. Symlinks outside the chroot will not work, but hardlinks do. I ended up solving this problem by hardlinking statically compiled basic programs under each user's home directory, under /bin and the creating a symlink from /usr/bin to /bin. The advantage of this approach is that you have very fine control of which programs each user has in their shell. The disadvantage is the increased disk space usage.
You'll need the MySQL client inside the chroot in order to allow access to the MySQL server, which is in it's own subsystem.
I'm doing this on a FreeBSD 5.2-RELEASE testing environment. I wanted to use the 5.x series mostly because they offer better jail support than the 4.x series, and due to the fact that NSS modules exist only in FreeBSD 5.0 and later. This is how I did this on my testing environment, and does not neccessarily reflect the setup my production environment runs.
My testing environment has two 80GB IDE hard drives, ad0 is a Seagate (156296322 blocks) and ad2 a Samsung ( blocks). I will create only one slice on each.
I'll now create a slightly modified setup of my 'A bootstrapped RAID-1 setup using Vinum' document. I will go through the installation pretty shortly here, see my Vinum document for more information about Vinum, in case you aren't familiar with it. Setting up a RAID-1 array is by no means neccessary for a working setup, but it adds a whole lot of redundancy.
Below is the partitioning scheme I picked when installing FreeBSD:
Part Mount Size ---- ----- ---- ad0s1b swap 2000MB (4096000 blocks) ad0s1a / 4000MB (8192000 blocks) ad0s1d /usr 2000MB (4096000 blocks) ad0s1e /tmp 2000MB (4096000 blocks) ad0s1f /var/log 2000MB (4096000 blocks) ad0s1g /var rest of drive (in my case 131720322 blocks, i.e. about 64347MB). ad2s1b swap 2000MB (4096000 blocks) ad2s1d /roootback 4000MB (8192000 blocks) ad2s1e /usr2 2000MB (4096000 blocks) ad2s1f /tmp2 2000MB (4096000 blocks) ad2s1g /var/log2 2000MB (4096000 blocks) ad2s1h /var2 131720322 blocks, i.e. about 64347MB
As my ad2 was slightly bigger, I left 64260 blocks unallocated (about 31MB), as I couldn't think of any reasonable use for a partition that small.
After the installation and loader configuration, I edited the ad0s1 bsdlabel in a similar fashion:
After that, I created a similar configuration file for Vinum:
drive Redrum device /dev/ad0s1h volume root plex org concat sd len 8192000s driveoffset 4095984s drive Redrum volume swap plex org concat sd len 4095719s driveoffset 265s drive Redrum volume usr plex org concat sd len 4096000s driveoffset 12287984s drive Redrum volume tmp plex org concat sd len 4096000s driveoffset 16383984s drive Redrum volume varlog plex org concat sd len 4096000s driveoffset 20479984s drive Redrum volume var plex org concat sd len 131720322s driveoffset 24575984s drive Redrum
After I loaded the file into Vinum, I edited /etc/fstab to the following and rebooted to single-user mode:
# Device Mountpoint FStype Options Dump Pass# /dev/vinum/swap none swap sw 0 0 /dev/vinum/root / ufs rw 1 1 /dev/vinum/usr /usr ufs rw 2 2 /dev/vinum/tmp /tmp ufs rw 2 2 /dev/vinum/varlog /var/log ufs rw 2 2 /dev/vinum/var /var ufs rw,userquota,groupquota 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
Edited bsdlabel for ad2s1 to match ad0s1 and created a config file for Vinum:
drive BlackRain device /dev/ad2s1h volume root plex org concat sd len 8192000s driveoffset 4095984s drive BlackRain volume swap plex org concat sd len 4095719s driveoffset 265s drive BlackRain volume usr plex org concat sd len 4096000s driveoffset 12287984s drive BlackRain volume tmp plex org concat sd len 4096000s driveoffset 16383984s drive BlackRain volume varlog plex org concat sd len 4096000s driveoffset 20479984s drive BlackRain volume var plex org concat sd len 131720322s driveoffset 24575984s drive BlackRain
Loaded config into Vinum, revived plexes and exited to multi-user mode.
The stock FreeBSD kernel does not include all of the features we need, and includes a few we don't. Thus, we need to compile a custom kernel. We'll definitely want to enable quota support:
The following option prevents something known as OS fingerprinting, which is a scan technique used to determine the type of operating system running on a host:
However, as I'm running a web server I will NOT include this option in the kernel, as suggested in the FreeBSD handbook. Another interesting option is:
This option enables ICMP error response bandwidth limiting. You typically want this option as it will help protect the machine from denial of service packet attacks. However, this option is enabled by default in FreeBSD 5.x, so no need to add that. The following is almost verbatim from the FreeBSD handbook. Feel free to view my kernel configuration file, called ZEUS, which is named after my testing environment's hostname.
I rebooted in order to load the new kernel.
All of the subsystem IP addresses are aliases to my server's first NIC fxp0 (Intel EtherExpress Pro/100+). I'm using the A-class internal IP address range 10.x.x.x. I'll refer to the actual FreeBSD installation which contains the jailcells as the 'host system'. I will not go into the routing of these addresses here, as that's somewhat irrelevant to the jail setup. However, below is a simple example.
My host system is at 10.0.0.5, netmask 255.0.0.0, and has a direct connection to the Internet through fxp0. Before adding any jails, my network setup looks like this:
<*>Internet<*> ---- <Router> ---- <fxp0 - 10.0.0.5>[FreeBSD 5.1]
Jails are IP-centric, so we need to add one IP for each jail we want to use. Because each jail requires its own IP address, the services on your box must be configured to listen to specific addresses, not just every available address. For example, SSHd listens on all addresses by default. You need to setup ListenAddress in the host system's /etc/ssh/sshd_config to whatever your host system's address is. I set mine to 10.0.0.5. Do a killall -HUP sshd to make sure SSHd notices the change in the config. If you fail to do this, conflicts may occur over the aliased IP address.
It is recommended that you experiment and caution that it is a lot easier to start with a 'fat' jail and remove things until it stops working, than it is to start with a 'thin' jail and add things until it works.
Here, I'm adding four IP addresses, which will be routed for Internet access in my router. In the host system's /etc/rc.conf I defined the following:
ifconfig_fxp0="inet 10.0.0.5 netmask 255.0.0.0" ifconfig_fxp0_alias0="inet alias 10.0.1.0 netmask 0xffffffff" ifconfig_fxp0_alias1="inet alias 10.0.1.1 netmask 0xffffffff" ifconfig_fxp0_alias2="inet alias 10.0.1.2 netmask 0xffffffff" ifconfig_fxp0_alias3="inet alias 10.0.1.3 netmask 0xffffffff"
Note that we need to use a netmask that doesn't coflict with our current netmask, as "aliases on the same subnet as the primary IP should always have a netmask of 255.255.255.255". When setting the mask to 255.255.255.255, routes will be automatically set up for the aliases. Setting the netmask to something might work, but probably not. After adding the aliases, my netowrk schematic would look like this:
<*>Internet<*> ---- <Router> ---- <fxp0 - 10.0.0.5>[FreeBSD 5.1] <fxp0> | | <Jail>10.0.1.0<Jail> <Jail>10.0.1.1<Jail> <Jail>10.0.1.2<Jail> <Jail>10.0.1.3<Jail>
One might go as far as to create the jail on a different subnet, and placing a firewall in between. However, I will not do so here.
I have written a small shell script for jail setup. It's almost verbatim from man jail(8), with a few additions I've found useful. The script takes one argument, the name of the jail to be created.
Simple, isn't it? Now, in order to avoid having to go through the whole make world process for each jail, I simply copied the newly created, 'fresh' jail as many times as I needed jails.
We need to make some changes and additions to the host system's /etc/rc.conf.
enable_quotas="YES" check_quotas="YES" kern_securelevel_enable="YES" clear_tmp_enable="YES" update_motd="NO" usbd_enable="NO" rpcbind_enable="NO" sendmail_enable="NO" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO"
RPCBind is evil. As I'm going to run Exim, no need for Sendmail to start (at all, see 'man rc.sendmail'). Note that the sendmail_enable="NONE" method is DEPRECATED. ICMP redirects can be used to launch a DOS (Denial Of Service) attack. Let's prevent that:
If you wish to start all of your jails automatically at boot time, add rows such as this:
jail_enable="YES" jail_list="kiwi cherry" jail_kiwi_hostname="kiwi" jail_kiwi_ip="10.0.1.0" jail_kiwi_rootdir="/var/jail/10.0.1.0" jail_kiwi_exec="/bin/sh /etc/rc" jail_cherry_hostname="cherry" jail_cherry_ip="10.0.1.1" jail_cherry_rootdir="/var/jail/10.0.1.1" jail_cherry_exec="/bin/sh /etc/rc"
Next, edit the host system's /etc/sysctl.conf, and place the following variables in it:
security.jail.set_hostname_allowed=0 security.jail.sysvipc_allowed=1 security.bsd.see_other_uids=0
The first option will prevent the superuser in the jail to change the hostname of the jail. The second option must unfortunately be activated, as PostgreSQL requires it. If it's set to 0, PostgreSQL cannot be run iside a jail. Jail-NG would offer a solution for this, but is not yet quite production-ready. The third options prevents users from seeing information about processes that are being run under another UID. I see this as a good thing.
I will use the host system's ports tree along with mount_nullfs to install ports inside the jails. But first, we need to update the ports tree. I executed the following commands on the host system to accomplish this. This is almost verbatim from the FreeBSD handbook.
Feel free to view the ports-supfile I used.
I'm going to setup MSNTP (a Simple Network Time Protocol daemon/client) in order to keep the host system and some of the other computers on my LAN synchronized. IMHO full NTP is FAR too bloated nowadays. Do NOT use the NTP servers I use in my example. Find ones closer to where you are located. On the host system I executed the following commands:
The idea here is to run the first msntp as the client to an external NTP server, and the second as a local server, which the other computers on my LAN will use. You might think "Hey, why not just run clients to external NTP sources on all of the other computers"? No. This is the proper way to do it.
Note that we needn't synchronize jails, as they use the host system's clock. It's sufficient the run 'tzsetup' in a jail to set the timezone once, and then see to it that the host system's clock is synchronized.
To start the msntp daemon at boot, enter the following options in the host system's /etc/rc.conf:
ntpdate_enable="YES" ntpdate_program="/usr/local/bin/msntp" ntpdate_flags="-S &"
Insert the following in the host system's crontab for root:
15 0,12 * * * /bin/nice --10 /usr/local/bin/msntp -r -x 480 ntp.hut.fi tick.keso.fi
I tried running the client msntp with the -a option, as recommended, but it failed. Don't really know why, maybe adjtime is broken? One could be paranoid and run the daemon in a jail of it's own, but I considered this setup safe enough, as my firewall blocks traffic from outside my LAN to the msntp daemon.
I wanted a more fine-grained control over my log files, and the ability to store all log files form the jails outside the jails, in a centralized loaction. This approach prevents log file tampering, in case a malicious user would gain access to the log files inside a jail. Having all log files on a separate partition also helps dealing with log-owerflow attacks.
The use of stunnel creates a SSL encrypted tunnel over which the log files pass between the log host and the client. Stunnel can encrypt any connection between two machines that are transferred using TCP. Because the original syslog can't send their logs to a remote log host over TCP, syslog-ng is used because it allows for TCP transfer.
If the remote log connection isn't encrypted, a packet sniffer, such as ethereal, can be used to read the log files as they are passed without encryption. Valuable information can be gathered by sniffing these log files that might pose a security risk.
Next, grab the syslog-ng package:
Now we need to replace FreeBSD's native syslogd with syslog-ng. This is done by placing the following in the host system's /etc/rc.conf:
Configuration of syslog-ng needs to be done carefully.
Before installing libnss-mysql, you need to have MySQL running. See my 'Jailing MySQL on FreeBSD' document for a description on how I did it. libnss-mysql will replace the current /etc/passwd and /etc/group.
The real problem arises when we wish to use quotas. In order for quotas to work, we need to match the host system's UIDs/GIDs to the ones in our jails (they must be exactly the same!). Here libnss-mysql comes to the rescue, as using it it's quite trivial to implement a unified user database for the host system and the jails. However, we will need to install libnss-mysql in all of the jails in which we wish to use quota.
On the host system, I executed the following commands:
Edit /etc/libnss-mysql.cfg and optionally /etc/libnss-mysql-root.cfg. We need to tell libnss-mysql where to find our MySQL server. In /etc/libnss-mysql.cfg, define:
host 10.0.1.0 database auth username nssuser password goodsecret ssl 1 timeout 3 compress 1
This will allow us to transmit compressed SQL data over a SSL connection, which should be pretty hard to sniff. Save the file and exit the editor. Next, edit /etc/nsswitch.conf and add 'mysql' to passwd/group:
passwd: files mysql group: files mysql
Also check the root:wheel is the owner of these files. Jail yourself to the SQL cell and load the sample database:
Next, give SELECT privilege (enough for successful authentication) to the libnss user:
GRANT SELECT ON auth.* TO [email protected] IDENTIFIED BY 'goodsecret' REQUIRE SSL;
Note the IP address! Even though we are connecting from 10.0.0.5 (the host system), MySQL sees it as 'localhost', as it's an alias for fxp0. The rest of the command should be pretty self-explanatory.
We'll need to GRANT access from our webserver subsystem as well:
GRANT SELECT ON auth.* TO [email protected] IDENTIFIED BY 'anothergoodsecret' REQUIRE SSL;
Tip: The 'id' command is quite useful for testing the setup.
'make' will not work - you need 'gmake'.
About the author
I'm a millennial digital nomad and a seasoned IT professional with over 20 years of cross-industry experience, ready to supercharge your business. Drop me a note or read more about what I can do for you!