Apache, PHP 7, FPM & Let’s Encrypt TLS certificates on Debian Stretch

This WP blog is running on a small server accompanied by teamobsession.at, the freerunning & parcour team from my brother. The reasoning for WordPress is Divi, which let’s not-so-experienced users build awesome websites, including family members.

Lately I’ve seen many OOM killer messages which sacrificed the MariaDB in most cases. I investigated over the past months, collecting logs, performance metrics and so on.


Apache, PHP and FPM

At some point, MariaDB tuning wasn’t helpful for memory optimizations. Since this box evolved from Debian Wheezy to Jessie to Stretch, I realized that it may be just Apache and PHP, freed mind after a relaxed vacation. I’m not a friend of limiting connections or debugging the mpm_prefork module (great explanation though in this blog post).

a2query -m mpm_prefork

vim /etc/apache2/mods-enabled/mpm_prefork.conf

Recent times with Icinga Web 2 have shown how easy, fast and reliable PHP FPM works. The PHP module is not directly loaded into Apache’s memory and then forked into child processes. Instead, the Apache processes forward the requests via socket to the FPM daemon which just executes given tasks. More tuning tips in this blog post.

While I figured that my box still runs the EOL PHP5 versions (a good indicator for bad performance or leaks), I did the upgrade in two ways:

  • Go for PHP 7 (
  • Go for PHP FPM

WP needs the mbstring & xml PHP modules as otherwise xmlrpc with Jetpack Publicize breaks, Debug: “https://jetpack.com/support/debug/?url=https://www.legendiary.at”.

Debian makes this upgrade super easy with just a couple of CLI commands enabling this globally. “a2enmod” was btw the inspiration for “icinga2 feature enable” 😉

apt-get update
apt-get install php php-mysql php-mbstring php-xml

apt-get install php-fpm

a2enmod proxy_fcgi setenvif
a2enconf php7.0-fpm
systemctl restart apache2

In order to test this, put a test php file somewhere

vim /var/www/html/p.php

<?php echo phpinfo(); ?>

Open it in the browser to check the “Server API” string. Next, delete it immediately as it exposes internal data which are a security risk these days.

rm /var/www/html/p.php 


TLS with Let’s Encrypt

I’m following the Let’s Encrypt evolution closely but always put it back on the TODO list. Modern times prove us right to only use TLS, and so does Google with marking non-https connections as insecure. Recently I’ve learned that the Let’s Encrypt chain is fully trusted by all major root programs, so you don’t need to provide the full CA chain in your web server’s TLS certificate configuration anymore.

Follow these instructions for Debian Stretch, and enable the backports repository first. Hint: I don’t like polluting the sources.list with custom stuff, I just put the configuration into sources.list.d. Then just install the certbot package.

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.debian.org/debian stretch-backports main

apt-get update
apt-get install python-certbot-apache -t stretch-backports

Run the certbot CLI command and provide the request information. I’m just using the default and enforce HTTPS connections.

certbot --authenticator webroot --installer apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): michael.friedrich@...

Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
(A)gree/(C)ancel: a

Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
(Y)es/(N)o: n

Which names would you like to activate HTTPS for?
1: legendiary.at
2: web.legendiary.at
3: www.legendiary.at
4: teamobsession.at
5: www.teamobsession.at
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1 3 4 5
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for legendiary.at
http-01 challenge for www.legendiary.at
http-01 challenge for teamobsession.at
http-01 challenge for www.teamobsession.at
Input the webroot for legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for www.legendiary.at:
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for www.legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for teamobsession.at:
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for teamobsession.at: (Enter 'c' to cancel): /var/www/www.teamobsession.at/htdocs

Select the webroot for www.teamobsession.at:
1: Enter a new webroot
2: /var/www/www.teamobsession.at/htdocs
3: /var/www/www.legendiary.at/htdocs
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 2
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Created an SSL vhost at /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/apache2/sites-enabled/www.legendiary.at.conf to ssl vhost in /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Redirecting vhost in /etc/apache2/sites-enabled/www.teamobsession.at.conf to ssl vhost in /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

Congratulations! You have successfully enabled https://legendiary.at,
https://www.legendiary.at, https://teamobsession.at, and https://www.teamobsession.at

You should test your configuration at:

- Congratulations! Your certificate and chain have been saved at:
Your key file has been saved at:
Your cert will expire on 2018-11-17. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew *all* of
your certificates, run "certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

The certbot command also automatically installs certificate renewal, so nothing to worry about. One of the domains is not automatically detected, so I needed a small adjustment to fully support the CNAME from www.freerunningacademy.at.

certbot run -d www.teamobsession.at -d teamobsession.at -d www.freerunningacademy.at -d www.legendiary.at -d legendiary.at

Voilà 🙂


Since I am using the Google Drive backup for 4+ years now, I’ve just extended its script to include “/etc/letsencrypt” and “/etc/apache2”. Backups are important 🙂


Future Plans

At one time I might just put everything into containers and let them run on NETWAYS NWS. My feature request with Let’s Encrypt support still stands 🙂 The other one is an A+ rating which involves more TLS configuration fine tuning. But that’s something for another weekend after another vacation 🙂

Upgrade to Fedora 28: nss-pem does not belong to a distupgrade repository

Fedora 28 was released a while ago, and I do not immediately upgrade my workstation from past experience with external repositories needing to catch up.

Starting with

sudo dnf --refresh upgrade
sudo dnf system-upgrade download --refresh --releasever=28

led to this error.

 Problem: nss-pem-1.0.3-6.fc27.i686 has inferior architecture
  - nss-pem-1.0.3-6.fc27.x86_64 does not belong to a distupgrade repository
  - problem with installed package nss-pem-1.0.3-6.fc27.i686

This is coming from steam which is installed via UnitedRPM/RPMFusion repository. There seems to be a package dependency change during the F27 cycle, where right now there’s no explicit dependency or provider is there.

sudo dnf remove nss-pem-1.0.3-6.fc27.i686
Abhängigkeiten sind aufgelöst.
 Paket                                     Arch                        Version                                   Paketquelle                                       Größe
 nss-pem                                   i686                        1.0.3-6.fc27                              @updates                                          215 k
Removing dependent packages:
 libcurl                                   i686                        7.55.1-10.fc27                            @updates                                          581 k
 libdbusmenu-gtk2                          i686                        16.04.0-4.fc27                            @fedora                                            79 k
 libdbusmenu-gtk3                          i686                        16.04.0-4.fc27                            @fedora                                            79 k
 nss                                       i686                        3.36.1-1.0.fc27                           @updates                                          2.4 M
 openldap                                  i686                        2.4.45-4.fc27                             @updates                                          1.0 M
 steam                                     i686                                        @rpmfusion-nonfree-updates                        2.7 M

This bug report and this blog post helped a lot, just pre-install the F28 package prior to the upgrade. In my case I had to force the package update.

sudo dnf update nss-pem --releasever=28 --best --allowerasing

Next, proceed with the upgrade again.

sudo dnf system-upgrade download --refresh --releasever=28

Say hi to Fedora 28 🙂

Clear redirect cache in Chrome

Chrome caches 301 redirects in a certain way. This one caused problems when testing redirects with monitoring-portal.org with / to /woltlab (now removed).

There is no direct configuration setting for this, you need to visit chrome://net-internals/ in Chrome. Then right click on the cursor icon in the right upper corner and select “clear cache”.

A taste of Austria: 5 years Germany & NETWAYS

Time really flies. I still remember my decision to move from Vienna to Nuremberg, like it was yesterday. Lately we’ve been talking about job application interviews at NETWAYS and I made my usual joke “Meet with Bernd, have a coffee and say yes”. I’ve never shown my CV or wrote an application letter.

It was and is a matter of trust and honesty. Bernd invited me to this adventure, and always has guided me when times were hard, or not so super exciting. He’s also the one who made me a more friendly, and calm person with a vision to lead and care about others. Just awesome.

Back at that time I also met Markus who helped me with the move into my Nuremberg located flat. Just 15 minutes walk to NETWAYS, not so far away from Anwanden. Markus and Nicole are the ones where I found my “second” family here in Nuremberg, far away from home. I really feel safe and relaxed when I’m with them. Life is good.

NETWAYS is family and friends. That includes bitching, arguing, joining the conversation and feeling so young when you are going somewhere. Like a trip with your school or student mates. We all have a vision at work – the most important thing is that we do care about each other. You’ll see that when a party is ending, cleaning up together. You’ll recognize that during a conference when everyone pushes the team to the next level. You can feel it, when a small group of crazy Germans, Austrians and Italians moves to a foreign country and is just pure natural and crazy. No-one is left behind and we all go the flow. Just friends.

Speaking of Austria, I tend to visit Linz more often these days. My CX-5 provides me the luxury to just do so, while travelling by train is somewhat “too much stress and plans”. And of course, I found my love with bringing the good Austrian stuff to Nuremberg. Dragee Keksi, Linzer Torte, Manner, and much more. Always honoured and loved, already travelling the world. A taste of Austria.

I did not really change my Austrian dialect to German or Franconia even. At some point, I just don’t care and use “funny” words. Like “the other” Markus is doing with his “Wos? Jetzt glaub i’s oba!” (What? Now I believe it … or something like that). It always has been a good start into a conversation to discuss the different words and meanings of German and Austrian words. It doesn’t matter which country you belong these days, we think global.

Travelling the world with NETWAYS, and Icinga. Made friends and achieved my personal goal: Visit events and give talks about the things I love. I’m also preferring English everywhere I can.  Open source unites.

I’ve seen different cultures, used to learn and communicate. Keeping calm if a discussion is on the run. My past me was reacting emotional when someone said “annoying” or “bad project” about “my” Icinga. These days I see the success of Icinga, how many different people are involved in it, with their different visions and the idea to push each other to the next level. Time to fade away and lean back.

I’ve also started to look into “new” things. Some points in the past have proven that I am running directly into a burn-out. This is the time when I get ill, which I normally don’t. I learned from projects which were huge, and failed. Failure is allowed at NETWAYS, and I am glad that I get the time and resources to keep calm, motivate again, and find my true vision. Lately “Icinga is running on the ISS” just nailed it again. Trust and motivation.

Managing projects and leading teams is a hard job, still my ongoing goal. Be it for Icinga or NETWAYS. I’m not the “perfect” coder, I like to organize things, design and create concepts, write documentation, do support, install all the things, and integrate them with other tools in the stack. There’s also Request Tracker as one of my new projects. And a new software is coming to life soon, all helped with the nice startup days last week at NETWAYS. Most recently I’ve also started to develop Icinga Web 2 modules. Finding new areas of expertise is pure motivation.

Probably I’m doing too many side projects, like Dashing, Vagrant, Plugins and also providing monitoring-portal.org as a community platform. Still, I learned to keep it in the 40 hours a week, and put aside everything when turning on the PS4 (I really love Assassin’s Creed Origins), building LEGO models, or just reading Perry Rhodan, still at 1170. Life is too short to work too much.

Friends married this year. The not-so-typical American-Italian-German wedding in lovely Nuremberg, many days full of love and joy. And the one you only do once in your life: Going to Kosovo, meet a new culture and enjoy life. Next year, some “round” birthdays are coming, and we’ll see where we’re heading. Our “we play offline games and drink a lot” round with Vanessa, Markus and Martin is also in the making again. Family and friends matter.

After all, it is an incredible journey and I don’t want to miss it. I love going to work and see my friends and colleagues. I love going back to Austria and visit my family. I love Nuremberg, Franconia and the country side. I love meeting friends and discussing the latest (LEGO) stuff, or just sitting in the garden having a G&T doing nothing. Tomorrow we’ll party hard at the NETWAYS XMas event, I’ll promise 🙂

Life is good with a taste of Austria 🙂

VLC on Fedora with Wayland: Big Icons, small text problem

I’ve always used VLC as my favorite video player on Linux. Recently they changed releases to 3.0 from Git in Fedora 25 (RPMFusion repository). This also included changes to work with Wayland instead of Xorg. Unfortunately the user interface was broken then – big icons, small text. Just looked like 800×600 on a full HD resolution, 27″ here.

Options described on the net where to create a custom skin, or clear the configuration cache. None of these worked unfortunately.

While looking for a possible bug I’ve found this issue which lead me to a new repository called “United RPMs”.

Thought I’d give it a try, since this issue proposes updated packages which fix the issue entirely.

sudo -rpm --import https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/URPMS-GPG-PUBLICKEY-Fedora-24
sudo dnf -y install https://github.com/UnitedRPMs/unitedrpms/releases/download/6/unitedrpms-$(rpm -E %fedora)-6.fc$(rpm -E %fedora).noarch.rpm

sudo dnf makecache

In order to prefer UnitedRPMs over RPMFusion, I’m explicitly setting the repositories on install (I don’t want to fiddle with yum priorities here).

sudo dnf remove vlc

sudo dnf install --repo=unitedrpms --repo=fedora vlc

Voilá, VLC works again.