Apache, PHP 7, FPM & Let’s Encrypt TLS certificates on Debian Stretch

This WP blog is running on a small server accompanied by teamobsession.at, the freerunning & parcour team from my brother. The reasoning for WordPress is Divi, which let’s not-so-experienced users build awesome websites, including family members.

Lately I’ve seen many OOM killer messages which sacrificed the MariaDB in most cases. I investigated over the past months, collecting logs, performance metrics and so on.

 

Apache, PHP and FPM

At some point, MariaDB tuning wasn’t helpful for memory optimizations. Since this box evolved from Debian Wheezy to Jessie to Stretch, I realized that it may be just Apache and PHP, freed mind after a relaxed vacation. I’m not a friend of limiting connections or debugging the mpm_prefork module (great explanation though in this blog post).

a2query -m mpm_prefork

vim /etc/apache2/mods-enabled/mpm_prefork.conf

Recent times with Icinga Web 2 have shown how easy, fast and reliable PHP FPM works. The PHP module is not directly loaded into Apache’s memory and then forked into child processes. Instead, the Apache processes forward the requests via socket to the FPM daemon which just executes given tasks. More tuning tips in this blog post.

While I figured that my box still runs the EOL PHP5 versions (a good indicator for bad performance or leaks), I did the upgrade in two ways:

  • Go for PHP 7 (
  • Go for PHP FPM

WP needs the mbstring & xml PHP modules as otherwise xmlrpc with Jetpack Publicize breaks, Debug: “https://jetpack.com/support/debug/?url=https://www.legendiary.at”.

Debian makes this upgrade super easy with just a couple of CLI commands enabling this globally. “a2enmod” was btw the inspiration for “icinga2 feature enable” 😉

apt-get update
apt-get install php php-mysql php-mbstring php-xml

apt-get install php-fpm

a2enmod proxy_fcgi setenvif
a2enconf php7.0-fpm
systemctl restart apache2

In order to test this, put a test php file somewhere

vim /var/www/html/p.php

<?php echo phpinfo(); ?>

Open it in the browser to check the “Server API” string. Next, delete it immediately as it exposes internal data which are a security risk these days.

rm /var/www/html/p.php 

 

TLS with Let’s Encrypt

I’m following the Let’s Encrypt evolution closely but always put it back on the TODO list. Modern times prove us right to only use TLS, and so does Google with marking non-https connections as insecure. Recently I’ve learned that the Let’s Encrypt chain is fully trusted by all major root programs, so you don’t need to provide the full CA chain in your web server’s TLS certificate configuration anymore.

Follow these instructions for Debian Stretch, and enable the backports repository first. Hint: I don’t like polluting the sources.list with custom stuff, I just put the configuration into sources.list.d. Then just install the certbot package.

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.debian.org/debian stretch-backports main
EOF

apt-get update
apt-get install python-certbot-apache -t stretch-backports

Run the certbot CLI command and provide the request information. I’m just using the default and enforce HTTPS connections.

certbot --authenticator webroot --installer apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): michael.friedrich@...

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: a

-------------------------------------------------------------------------------
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
-------------------------------------------------------------------------------
(Y)es/(N)o: n

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: legendiary.at
2: web.legendiary.at
3: www.legendiary.at
4: teamobsession.at
5: www.teamobsession.at
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1 3 4 5
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for legendiary.at
http-01 challenge for www.legendiary.at
http-01 challenge for teamobsession.at
http-01 challenge for www.teamobsession.at
Input the webroot for legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for www.legendiary.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for www.legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for teamobsession.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for teamobsession.at: (Enter 'c' to cancel): /var/www/www.teamobsession.at/htdocs

Select the webroot for www.teamobsession.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.teamobsession.at/htdocs
3: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 2
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Created an SSL vhost at /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/apache2/sites-enabled/www.legendiary.at.conf to ssl vhost in /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Redirecting vhost in /etc/apache2/sites-enabled/www.teamobsession.at.conf to ssl vhost in /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://legendiary.at,
https://www.legendiary.at, https://teamobsession.at, and https://www.teamobsession.at

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=legendiary.at
https://www.ssllabs.com/ssltest/analyze.html?d=www.legendiary.at
https://www.ssllabs.com/ssltest/analyze.html?d=teamobsession.at
https://www.ssllabs.com/ssltest/analyze.html?d=www.teamobsession.at
-------------------------------------------------------------------------------

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/legendiary.at/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/legendiary.at/privkey.pem
Your cert will expire on 2018-11-17. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew *all* of
your certificates, run "certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

The certbot command also automatically installs certificate renewal, so nothing to worry about. One of the domains is not automatically detected, so I needed a small adjustment to fully support the CNAME from www.freerunningacademy.at.

certbot run -d www.teamobsession.at -d teamobsession.at -d www.freerunningacademy.at -d www.legendiary.at -d legendiary.at

Voilà 🙂

Backups

Since I am using the Google Drive backup for 4+ years now, I’ve just extended its script to include “/etc/letsencrypt” and “/etc/apache2”. Backups are important 🙂

 

Future Plans

At one time I might just put everything into containers and let them run on NETWAYS NWS. My feature request with Let’s Encrypt support still stands 🙂 The other one is an A+ rating which involves more TLS configuration fine tuning. But that’s something for another weekend after another vacation 🙂

Virtualbox missing vboxdrv dkms modules

During the one of last d-u some trigger accidentally removed the linux-headers meta package which is necessary for building dynamic kernel modules for nvidia and virtualbox.

Start-Date: 2014-06-07  21:17:14
Commandline: apt-get dist-upgrade
Install: python3-markupsafe:amd64 (0.23-1, automatic), gir1.2-secret-1:amd64 (0.18-1, automatic), python3-mako:amd64 (0.9.1-1, automatic), libgcrypt20:
amd64 (1.6.1-2, automatic)
Upgrade: libtotem-plparser18:amd64 (3.10.2-1, 3.10.2-3), rhythmbox-plugins:amd64 (3.0.1-1+b2, 3.0.3-1+b1), kde-runtime:amd64 (4.12.4-1, 4.13.1-1), rhyt
hmbox-data:amd64 (3.0.1-1, 3.0.3-1), rhythmbox-plugin-cdrecorder:amd64 (3.0.1-1+b2, 3.0.3-1+b1), libssh2-1:amd64 (1.4.3-2, 1.4.3-3), rhythmbox:amd64 (3
.0.1-1+b2, 3.0.3-1+b1), plasma-scriptengine-javascript:amd64 (4.12.4-1, 4.13.1-1), gir1.2-rb-3.0:amd64 (3.0.1-1+b2, 3.0.3-1+b1), librhythmbox-core8:amd
64 (3.0.1-1+b2, 3.0.3-1+b1)
Remove: linux-headers-3.13-1-amd64:amd64 (3.13.10-1), linux-headers-3.14-1-amd64:amd64 (3.14.4-1), linux-compiler-gcc-4.8-x86:amd64 (3.14.4-1), linux-headers-amd64:amd64 (3.14+57)
End-Date: 2014-06-07  21:17:26

Virtualbox does start but failed with a fancy non-telling error.

virtualbox_kernel_module_missing

At first sight I was confused with ‘/etc/init.d/vboxdrv setup’ which does not exist with systemd anymore.
Reinstalling the Virtualbox DKMS package just told me – ah, there’s something missing.

# apt-get install --reinstall virtualbox-dkms
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
0 aktualisiert, 0 neu installiert, 1 erneut installiert, 0 zu entfernen und 1 nicht aktualisiert.
Es müssen noch 0 B von 559 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 0 B Plattenplatz zusätzlich benutzt.
(Lese Datenbank ... 297911 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../virtualbox-dkms_4.3.12-dfsg-1_all.deb ...

------------------------------
Deleting module version: 4.3.12
completely from the DKMS tree.
------------------------------
Done.
Entpacken von virtualbox-dkms (4.3.12-dfsg-1) über (4.3.12-dfsg-1) ...
virtualbox-dkms (4.3.12-dfsg-1) wird eingerichtet ...
Loading new virtualbox-4.3.12 DKMS files...
Building only for 3.14-1-amd64
Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.

Fix it by installing the kernel headers and reinstalling the virtualbox dkms package.

# apt-get install linux-headers-amd64
# apt-get install --reinstall virtualbox-dkms

Then the vboxdrv kernel module has to be loaded.

# modprobe vboxdrv

Try to boot the vm, it will fail with the network interfaces requiring the additional ‘vboxnetflt’ kernel module.

# modprobe vboxnetflt

From sysvinit to systemd in Debian Jessie

sysvinit-core gets removed on dist-upgrade, and systemd-sysv is installed instead. Therefore the dependency for sysvinit is fulfilled and a smooth transition to systemd is ensured in current Debian Jessie.

nbmif ~ # apt-get dist-upgrade
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Die folgenden Pakete wurden automatisch installiert und werden nicht mehr benötigt:
  libqmi-glib0 xulrunner-29
Verwenden Sie »apt-get autoremove«, um sie zu entfernen.
Fertig
Die folgenden Pakete werden ENTFERNT:
  sysvinit-core
Die folgenden NEUEN Pakete werden installiert:
  libqmi-glib1 libqmi-proxy systemd-sysv
Die folgenden Pakete werden aktualisiert (Upgrade):
  libmm-glib0 libpam-systemd libsystemd-daemon0 libsystemd-journal0 libsystemd-login0 modemmanager systemd
7 aktualisiert, 3 neu installiert, 1 zu entfernen und 0 nicht aktualisiert.
Es müssen 2.294 kB an Archiven heruntergeladen werden.
Nach dieser Operation werden 1.825 kB Plattenplatz zusätzlich benutzt.
Möchten Sie fortfahren? [J/n]

The previous init binary is now a symlink to systemd.

nbmif ~ # ls -la /sbin/init
lrwxrwxrwx 1 root root 20 Jun 28 13:28 /sbin/init -> /lib/systemd/systemd*

Reboot – thanks to the parallel tasks it’s a matter of seconds on a i5 with 8gb ram and a samsung evo ssd.

Remove trailing whitespaces on save in Komodo Edit 8

Komodo Edit 8 is a free editor (not using the commercial IDE) but is sometimes hard to configure. It certainly plays well while hacking Icinga 2 in C++ or Icinga Web 2 in PHP.

On a fresh install, the trailing whitespaces are not removed when saving a document (and it really should, they are annoying when you open it with vim/git diff and whitespaces highlighting enabled).

Navigate to Edit – Preferences – Editor – Save Options and tick Clean trailing whitespaces and EOL markers and Only clean changed lines. The last option saves us from cleaning the entire document and generate horrible git diffs solving whitespaces issues caused by others. (i hate git diffs solving everything but hiding the real fixed code diff).

komodo_edit_clean_trailing_whitespaces_on_save

Debian Jessie, Chromium 35, NPAPI, Aura and the flash plugin

Chromium 35 did bite me with flashplugin-nonfree not being supported anymore – on saturday with a fresh install of Debian Testing, and later after dist-upgrade on my workstation fetching the latest version.

# less /var/log/apt/history.log

Start-Date: 2014-06-07  21:07:20
Commandline: apt-get upgrade
...
chromium:amd64 (34.0.1847.116-1~deb7u1, 35.0.1916.114-2),
...
michi@imagine ~ $ dpkg -l *chromium* | grep ^ii
ii  chromium                     35.0.1916.114-2 amd64        Chromium web browser
ii  chromium-inspector           35.0.1916.114-2 all          page inspector for the Chromium browser

michi@imagine ~ $ dpkg -l *flashplugin* | grep ^ii
ii  flashplugin-nonfree 1:3.4        amd64        Adobe Flash Player - browser plugin

which results in a fancy browser warning:

debian_testing_flash_player

 

 

The reason is simple – Chrome developers decided to drop/remove support for NPAPI plugins in Chrome 35 changing to Linux Aura, rendering the flash plugin incompatible.

Luckily there’s an alternate plugin around already, and explained on the Debian wiki, the pepper flash player.

# apt-get install pepperflashplugin-nonfree

This wrapper package downloads the current Chrome Debian package from google.com and unpacks the pepper flash plugin due to license issues with redistribution of that plugin (hooray, yet again).

Chromium will primarly detect that flash plugin, and starting to work again (after closing and restarting it). I’m aware of the fact that there’s gnash and other alternatives, but they either did not work or caused too much (compatibility) troubles.

Looks far better now 🙂

debian_testing_flash_player_fixed

Gnome 3: Change default user dirs

Presumingly somewhere hidden between gnome-session-properties, gnome-tweak-tool and dconf-editor, but that’s the easierst way:

$ vim ~/.config/user-dirs.dirs

XDG_DESKTOP_DIR="$HOME/Desktop"
XDG_DOWNLOAD_DIR="$HOME/"
XDG_TEMPLATES_DIR="$HOME/"
XDG_PUBLICSHARE_DIR="$HOME/"
XDG_DOCUMENTS_DIR="$HOME/"
XDG_MUSIC_DIR="$HOME/"
XDG_PICTURES_DIR="$HOME/"
XDG_VIDEOS_DIR="$HOME/"

Mainly for the reason that upgrades might re-create the nastly default dirs.