New Server!
Posted on .
So I went and ordered myself a new server. My old one was a VPS from Linode with 1 core, 1 GB of RAM and a 24 GB disk. The new one is a dedicated server from online.net with 8 cores, 8 GB of RAM and 1 TB of hard disk space. At the same time it is only slightly more expensive so I jumped at the opportunity. How reliable it actually is will only be shown with time, but I like living on the bleeding edge. So I thought I would write a blog post about all the stuff I run into when setting up the new server. Note: This post is meant for reference only, not as a guide. Be sure look for recommendations from people wiser than myself regarding any security settings.
Installing Arch the Hard Way
Speaking of bleeding edge, online.net's server manager did not have the option of installing Arch Linux on the box. There was also no way to mount an image of my choosing as a CD or USB disk. So, after a few initial hiccups, I installed Debian 8 on the server and proceeded to replace it with Arch Linux – while it was still running. It turned out to be quite a hardcore Linux adventure – at least on my scale – but eventually I got it to work. What I did was I first set up a bootstrap Arch install using arch-bootstrap in my home directory. This was to get the Arch Install Scripts that help when setting up an Arch install.
Next I turned off swap on the server, reformatted the swap partition as ext4 and installed on it the Arch system that would eventually replace the Debian install. I chrooted into it and installed what I needed, such as the syslinux bootloader, SSH server, tmux and other utilities. I needed to be extra careful with the syslinux setup, as the default setup would reference the swap partition instead of the main partition (where Debian was living at that time). If I referenced the wrong partition, the system would not be able to boot anymore.
When I had everything configured in the chroot, I first replaced the boot partition with the chroot's boot folder and
installed the syslinux MBR boot code. Then I replaced the Debian system by rsyncing files from the Arch partition into
the main partition. I used the --delete and --exclude switches to not move automatically created paths like
/proc/. After doing that I had to power cycle the machine, because all of my binaries stopped working (because some of
them were Debian binaries stored in memory with different library versions than what was now on disk). I waited in
anticipation and… it never responded to ping.
I logged into the virtual console and started debugging. Most programs, like curl and pacman, crashed when they were
started due to failures with dynamic libraries. This was because I had failed with my --delete switch a bit, leading
to Debian's dynamic libraries still existing and conflicting with Arch's binaries. I removed all the remaining Debian
stuff and everything started working.
SSH Configuration
There are many ways for securing SSH. Two often repeated suggestions are to run the SSH server on a nonstandard port and to only enable key authentication. The first one is security by obscurity and personally I'm not really into it. It also is inconvenient as you have to list the port in every tool that uses SSH. The second suggestion is not feasible, because I have other users that need simple password authentication. It would be quite inconvenient for me to carry my keys with me everywhere I went as well. I consider a combination of strong passwords and fail2ban good enough for this purpose.
I'm not a cryptographer, so when talking about ciphers, key exchange functions and all that stuff, I'm at the mercy of
people wiser than myself. For my SSH server, I basically followed
this guide for choosing good kex algorithms, ciphers
and MACs. What I ended up with is this sshd_config (I've only included the most relevant settings):
Protocol 2
HostKey /etc/ssh/ssh_host_ed25519_key # Ed25519 key preferred
HostKey /etc/ssh/ssh_host_rsa_key # 4096 bit RSA
# No DSA or ECDSA
LogLevel VERBOSE
PermitRootLogin no
MaxAuthTries 3
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com
I also created my own DH primes (moduli) to prevent precalculation by running the following:
ssh-keygen -G moduli-2048.candidates -b 2048
ssh-keygen -T moduli-2048 -f moduli-2048.candidates
mv moduli-2048 /etc/ssh/moduli
There are probably lots of other settings that I should have included too. If you know some essential setting that I did not mention, shoot me a comment.
One hidden benefit from these settings seems to be that botnets that try to bruteforce SSH passwords cannot seem to connect to the server anymore due to the modern algorithm choices. Reading the logs there have not been any password attempts so far. What I do see is a bunch of this:
Oct 31 21:40:10 octane sshd[22102]: Connection from 177.124.226.37 port 48636 on 62.210.76.20 port 22
Oct 31 21:40:10 octane sshd[22102]: Connection closed by 177.124.226.37 [preauth]
Oct 31 21:40:42 octane sshd[22110]: Connection from 43.229.53.48 port 39634 on 62.210.76.20 port 22
Oct 31 21:40:42 octane sshd[22110]: fatal: Unable to negotiate with 43.229.53.48: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 [preauth]
Oct 31 21:41:50 octane sshd[22127]: Connection from 177.124.226.37 port 39771 on 62.210.76.20 port 22
Oct 31 21:41:50 octane sshd[22127]: Connection closed by 177.124.226.37 [preauth]
Oct 31 21:43:29 octane sshd[22147]: Connection from 177.124.226.37 port 59133 on 62.210.76.20 port 22
Oct 31 21:43:30 octane sshd[22147]: Connection closed by 177.124.226.37 [preauth]
Note that these modern kex settings may be incompatible with different SSH clients. Personally I had to update my Cyberduck SFTP client to the latest beta to be able to connect. A bigger issue for some may be that these settings currently prevent PuTTY from connecting because it does not support the Curve 25519 kex algorithm and its support of the DH group exchange is broken. Curve 25519 is supported in the latest nightly builds but, at the time of writing, not in the latest stable version. This may affect other PuTTY based clients as well, at least PuTTY Tray.
Nginx and TLS
I use HTTPS for many of my domains, such as this blog. This time I wanted to configure HTTPS correctly. I used this guide and some googling. With HTTPS I also wanted to choose modern algorithms that would provide maximum security. Here are the settings I ended up with (again, only relevant settings are included):
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Disable SSLv3 to protect from POODLE
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# Update regularly!
# Safe cipher suite list from https://weakdh.org/sysadmin.html
# Also enables the use of our own DH group
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
ssl_dhparam 'dhparams.pem'; # Generated with `openssl dhparam -out dhparams.pem 2048`
# Additional HTTPS settings that should go to the server section of the specific virtual server
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
# Forbid HTTPS sites from being shown in a frame
add_header X-Frame-Options "DENY";
These settings have allowed me to get an A+ for this blog from the Qualys SSL test. Again, if you see something that I should consider in addition to those, let me know.
dibbler (DHCPv6)
My new server uses prefix delegation to get its IPv6 range. For this,
I needed to set up a DHCPv6 client. I ended going with dibbler, but, not being familiar
with prefix delegation beforehand, it took some trial and error (and asking around on IRC) to find out how it worked. In
the end what worked was this config, in addition to adding my DUID to
/var/lib/dibbler/client-duid:
# /etc/dibbler/client.conf
log-mode short
log-level 7
# inactive-mode allows dibbler to start even if network is not fully
# up yet
inactive-mode
# use the interface connected to your WAN
iface "eth0" {
ia
pd
option dns-server
}
Above, the ia means "request one address from the server" and the pd means prefix delegation. The inactive-mode
was key in getting dibbler to play well with Systemd. Without it, it would try to request before the network was up and
would die when it couldn't. In inactive mode it waits until the network is up before acting.
This setup gives me one IPv6 address from the range specified with the DUID. To add others, I use systemd-networkd with the following config file:
[Match]
Name=eth0
[Network]
DHCP=none
DNS=195.154.228.249
DNS=62.210.16.9
DNS=8.8.8.8
[Address]
Address=62.210.76.20
[Address]
Address=2001:0bc8:2082:01cd:abcd:abcd:abcd:abcd/128
[Address]
Address=2001:0bc8:2082:0123:4567:89ab:cdef:1/128
[Address]
Address=2001:0bc8:2082:0100:1:5ee:dead:c0de/128
[Route]
Gateway=62.210.76.1
Whenever I need a new IPv6 address, I add it to the pool above and run systemctl restart systemd-networkd.