Apache, PHP 7, FPM & Let’s Encrypt TLS certificates on Debian Stretch

This WP blog is running on a small server accompanied by teamobsession.at, the freerunning & parcour team from my brother. The reasoning for WordPress is Divi, which let’s not-so-experienced users build awesome websites, including family members.

Lately I’ve seen many OOM killer messages which sacrificed the MariaDB in most cases. I investigated over the past months, collecting logs, performance metrics and so on.

 

Apache, PHP and FPM

At some point, MariaDB tuning wasn’t helpful for memory optimizations. Since this box evolved from Debian Wheezy to Jessie to Stretch, I realized that it may be just Apache and PHP, freed mind after a relaxed vacation. I’m not a friend of limiting connections or debugging the mpm_prefork module (great explanation though in this blog post).

a2query -m mpm_prefork

vim /etc/apache2/mods-enabled/mpm_prefork.conf

Recent times with Icinga Web 2 have shown how easy, fast and reliable PHP FPM works. The PHP module is not directly loaded into Apache’s memory and then forked into child processes. Instead, the Apache processes forward the requests via socket to the FPM daemon which just executes given tasks. More tuning tips in this blog post.

While I figured that my box still runs the EOL PHP5 versions (a good indicator for bad performance or leaks), I did the upgrade in two ways:

  • Go for PHP 7 (
  • Go for PHP FPM

WP needs the mbstring & xml PHP modules as otherwise xmlrpc with Jetpack Publicize breaks, Debug: “https://jetpack.com/support/debug/?url=https://www.legendiary.at”.

Debian makes this upgrade super easy with just a couple of CLI commands enabling this globally. “a2enmod” was btw the inspiration for “icinga2 feature enable” 😉

apt-get update
apt-get install php php-mysql php-mbstring php-xml

apt-get install php-fpm

a2enmod proxy_fcgi setenvif
a2enconf php7.0-fpm
systemctl restart apache2

In order to test this, put a test php file somewhere

vim /var/www/html/p.php

<?php echo phpinfo(); ?>

Open it in the browser to check the “Server API” string. Next, delete it immediately as it exposes internal data which are a security risk these days.

rm /var/www/html/p.php 

 

TLS with Let’s Encrypt

I’m following the Let’s Encrypt evolution closely but always put it back on the TODO list. Modern times prove us right to only use TLS, and so does Google with marking non-https connections as insecure. Recently I’ve learned that the Let’s Encrypt chain is fully trusted by all major root programs, so you don’t need to provide the full CA chain in your web server’s TLS certificate configuration anymore.

Follow these instructions for Debian Stretch, and enable the backports repository first. Hint: I don’t like polluting the sources.list with custom stuff, I just put the configuration into sources.list.d. Then just install the certbot package.

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.debian.org/debian stretch-backports main
EOF

apt-get update
apt-get install python-certbot-apache -t stretch-backports

Run the certbot CLI command and provide the request information. I’m just using the default and enforce HTTPS connections.

certbot --authenticator webroot --installer apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): michael.friedrich@...

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v01.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: a

-------------------------------------------------------------------------------
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
-------------------------------------------------------------------------------
(Y)es/(N)o: n

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: legendiary.at
2: web.legendiary.at
3: www.legendiary.at
4: teamobsession.at
5: www.teamobsession.at
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1 3 4 5
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for legendiary.at
http-01 challenge for www.legendiary.at
http-01 challenge for teamobsession.at
http-01 challenge for www.teamobsession.at
Input the webroot for legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for www.legendiary.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for www.legendiary.at: (Enter 'c' to cancel): /var/www/www.legendiary.at/htdocs

Select the webroot for teamobsession.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Input the webroot for teamobsession.at: (Enter 'c' to cancel): /var/www/www.teamobsession.at/htdocs

Select the webroot for www.teamobsession.at:
-------------------------------------------------------------------------------
1: Enter a new webroot
2: /var/www/www.teamobsession.at/htdocs
3: /var/www/www.legendiary.at/htdocs
-------------------------------------------------------------------------------
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 2
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Created an SSL vhost at /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/apache2/sites-enabled/www.legendiary.at.conf to ssl vhost in /etc/apache2/sites-available/www.legendiary.at-le-ssl.conf
Redirecting vhost in /etc/apache2/sites-enabled/www.teamobsession.at.conf to ssl vhost in /etc/apache2/sites-available/www.teamobsession.at-le-ssl.conf

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://legendiary.at,
https://www.legendiary.at, https://teamobsession.at, and https://www.teamobsession.at

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=legendiary.at
https://www.ssllabs.com/ssltest/analyze.html?d=www.legendiary.at
https://www.ssllabs.com/ssltest/analyze.html?d=teamobsession.at
https://www.ssllabs.com/ssltest/analyze.html?d=www.teamobsession.at
-------------------------------------------------------------------------------

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/legendiary.at/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/legendiary.at/privkey.pem
Your cert will expire on 2018-11-17. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew *all* of
your certificates, run "certbot renew"
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

The certbot command also automatically installs certificate renewal, so nothing to worry about. One of the domains is not automatically detected, so I needed a small adjustment to fully support the CNAME from www.freerunningacademy.at.

certbot run -d www.teamobsession.at -d teamobsession.at -d www.freerunningacademy.at -d www.legendiary.at -d legendiary.at

Voilà 🙂

Backups

Since I am using the Google Drive backup for 4+ years now, I’ve just extended its script to include “/etc/letsencrypt” and “/etc/apache2”. Backups are important 🙂

 

Future Plans

At one time I might just put everything into containers and let them run on NETWAYS NWS. My feature request with Let’s Encrypt support still stands 🙂 The other one is an A+ rating which involves more TLS configuration fine tuning. But that’s something for another weekend after another vacation 🙂

Upgrade to Fedora 28: nss-pem does not belong to a distupgrade repository

Fedora 28 was released a while ago, and I do not immediately upgrade my workstation from past experience with external repositories needing to catch up.

Starting with

sudo dnf --refresh upgrade
sudo dnf system-upgrade download --refresh --releasever=28

led to this error.

Fehler: 
 Problem: nss-pem-1.0.3-6.fc27.i686 has inferior architecture
  - nss-pem-1.0.3-6.fc27.x86_64 does not belong to a distupgrade repository
  - problem with installed package nss-pem-1.0.3-6.fc27.i686

This is coming from steam which is installed via UnitedRPM/RPMFusion repository. There seems to be a package dependency change during the F27 cycle, where right now there’s no explicit dependency or provider is there.

sudo dnf remove nss-pem-1.0.3-6.fc27.i686
Abhängigkeiten sind aufgelöst.
=========================================================================================================================================================================
 Paket                                     Arch                        Version                                   Paketquelle                                       Größe
=========================================================================================================================================================================
Entfernen:
 nss-pem                                   i686                        1.0.3-6.fc27                              @updates                                          215 k
Removing dependent packages:
 libcurl                                   i686                        7.55.1-10.fc27                            @updates                                          581 k
 libdbusmenu-gtk2                          i686                        16.04.0-4.fc27                            @fedora                                            79 k
 libdbusmenu-gtk3                          i686                        16.04.0-4.fc27                            @fedora                                            79 k
 nss                                       i686                        3.36.1-1.0.fc27                           @updates                                          2.4 M
 openldap                                  i686                        2.4.45-4.fc27                             @updates                                          1.0 M
 steam                                     i686                        1.0.0.54-17.fc27                          @rpmfusion-nonfree-updates                        2.7 M

This bug report and this blog post helped a lot, just pre-install the F28 package prior to the upgrade. In my case I had to force the package update.

sudo dnf update nss-pem --releasever=28 --best --allowerasing

Next, proceed with the upgrade again.

sudo dnf system-upgrade download --refresh --releasever=28

Say hi to Fedora 28 🙂

Clear redirect cache in Chrome

Chrome caches 301 redirects in a certain way. This one caused problems when testing redirects with monitoring-portal.org with / to /woltlab (now removed).

There is no direct configuration setting for this, you need to visit chrome://net-internals/ in Chrome. Then right click on the cursor icon in the right upper corner and select “clear cache”.

VLC on Fedora with Wayland: Big Icons, small text problem

I’ve always used VLC as my favorite video player on Linux. Recently they changed releases to 3.0 from Git in Fedora 25 (RPMFusion repository). This also included changes to work with Wayland instead of Xorg. Unfortunately the user interface was broken then – big icons, small text. Just looked like 800×600 on a full HD resolution, 27″ here.

Options described on the net where to create a custom skin, or clear the configuration cache. None of these worked unfortunately.

While looking for a possible bug I’ve found this issue which lead me to a new repository called “United RPMs”.

Thought I’d give it a try, since this issue proposes updated packages which fix the issue entirely.

sudo -rpm --import https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/URPMS-GPG-PUBLICKEY-Fedora-24
sudo dnf -y install https://github.com/UnitedRPMs/unitedrpms/releases/download/6/unitedrpms-$(rpm -E %fedora)-6.fc$(rpm -E %fedora).noarch.rpm

sudo dnf makecache

In order to prefer UnitedRPMs over RPMFusion, I’m explicitly setting the repositories on install (I don’t want to fiddle with yum priorities here).

sudo dnf remove vlc

sudo dnf install --repo=unitedrpms --repo=fedora vlc

Voilá, VLC works again.

lldb NameError: name ‘run_one_line’ is not defined

I’m a heavy lldb user during Icinga 2 development. Most recently I got many of those error messages when starting lldb for debugging Icinga 2.

mbmif /usr/local/icinga2/etc/icinga2/tests (master) # lldb -- /usr/local/icinga2/lib/icinga2/sbin/icinga2 console
(lldb) target create "/usr/local/icinga2/lib/icinga2/sbin/icinga2"
Traceback (most recent call last):
  File "", line 1, in 
  File "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/__init__.py", line 98, in 
    import six
ImportError: No module named six
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'run_one_line' is not defined

Turns out that I have Python2 installed from a dependency in Homebrew. The lldb scripts just use the system path for determining the preferred Python binary.

A sensible workaround is discussed here:

$ /usr/local/bin/pip install six

Works again.

mbmif /usr/local/icinga2/etc/icinga2/tests (master) # lldb -- /usr/local/icinga2/lib/icinga2/sbin/icinga2 console
(lldb) target create "/usr/local/icinga2/lib/icinga2/sbin/icinga2"
Current executable set to '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64).
(lldb) settings set -- target.run-args  "console"
(lldb) q

Since I wanted to check which package requires python in /usr/local/bin/python I found this command

$ brew list | while read cask; do echo -n "$cask ->"; brew deps $cask | awk '{printf(" %s ", $0)}'; echo ""; done

I can’t get rid of pygtk and macvim, so each new install/update will pull Python again. The reason why they build their own Python is somewhat unsafe C++ functions. Guess I don’t want to dig any deeper here.

Homebrew Caskroom migration

I’m using Homebrew on my Macbook. It is a great addition to installing software when you are used to package managers from the Linux world.

There’s also an extension called Homebrew Cask which allows you to manage MacOS applications, such as Adium or Gimp. This saves you the hassle of manually downloading the package/dmg files and automates the installation/updates.

Lately when doing an update again, there was a notice about a changed Caskroom location.

michi@mbmif ~ $ brew cask list
Warning: The default Caskroom location has moved to /usr/local/Caskroom.
Please migrate your Casks to the new location and delete /opt/homebrew-cask/Caskroom,
or if you would like to keep your Caskroom at /opt/homebrew-cask/Caskroom, add the
following to your HOMEBREW_CASK_OPTS:
  --caskroom=/opt/homebrew-cask/Caskroom
For more details on each of those options, see https://github.com/caskroom/homebrew-cask/issues/21913.
apache-directory-studio  filezilla                gimp                     macvim                   vlc                      xquartz
bitbar                   firefox                  java7                    mysqlworkbench           wireshark

Ok, what would I do now? “Migration” could just mean moving the directories. But wait, the installed applications could be symlinked into ~/Applications instead of being moved (#13966).

Looking into the mentioned github issue #21913 shed some light on how to fix it. Moving the directories will make “brew cask list” shut up about the changed location, but later uninstalls will fail due to dangling symlinks. The solution is simple – force an installation again after moving the installed casks.

michi@mbmif ~ $ mv /opt/homebrew-cask/Caskroom /usr/local
michi@mbmif ~ $ brew cask list
apache-directory-studio  filezilla                gimp                     macvim                   vlc                      xquartz
bitbar                   firefox                  java7                    mysqlworkbench           wireshark

michi@mbmif ~ $ for cask in $(brew cask list); do brew cask install $cask --force; done

Done 🙂