If you are following this blog, you probably remember per-process namespaces. Today, I'm going to tell you how I did use them in the process of preparing this server to be upgraded to Lenny.
I must say I was not using the most recent stuff on this server, and this is why I needed such preparation. First, this server is still running with php4. And well, the following line in the apt-get dist-upgrade output got my full attention (emphasis mine):
The following packages will be REMOVED:
  cacti libapache2-mod-php4 libarchive-tar-perl libcurl3-openssl-dev
  libgssapi2 libpci2 librpm4 libssp0 linux-kernel-headers modutils php4-mysql
While this server is not important enough not to be broken for a couple hours, I do like to test procedures that could help on servers that are.
The first thing I needed to do on this server was obviously to upgrade php. But we all know how php applications are not fully compatible with all versions of php, so I also needed to test the upgrade was not breaking anything.
On a server you don't care much about, you can just upgrade, test if all is okay, and be done with it. Obviously, if all is not okay, your visitors will see it, and you may also have a bad time downgrading.
Another way to perform the upgrade is to have a similarly installed server on the side, test and validate the upgrade there, and replicate the upgrade on the production server if everything is fine. However, that does require additional resources, and possibly to setup them if they don't already exist.
A cheaper way to do the above is to do it on the production server, both in-place and on the side (you'll see what I mean), using unionfs and per-process namespaces. Full containers could be used instead of per-process namespaces (such as openvz, vserver or lxc), but they still require much more setup, especially when you don't use them in the first place. Chroots could work just as well as per-process namespaces, but one of the ideas here is to expose the per-process namespaces feature, and allow for improvement of this procedure with pid and network namespaces, which are not available in Etch (where I'm starting from), but are in Lenny.
Unionfs allows to merge several directories into a single one, accessing some read-only, and others read-write. Installing unionfs is as easy as running the following command:
apt-get install unionfs-modules-`uname -r`
I won't describe all the kinds of setups that are possible with unionfs, but only one typical use case, which is what we will be using here:
# mkdir /tmp/root-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt
The first thing we do here is to create an empty directory. Next, we merge it with the root filesystem (/), that we will keep read-only (meaning unionfs won't allow itself to write there), and mount the merged filesystem under /mnt. none could just be anything, as there is no device to be mounted.
The result, in /mnt, is just something that looks like the root filesystem:
# ls /
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var
# ls /mnt
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var
But creating or modifying a file will do so in /tmp/root-cow:
# echo a > /mnt/a
# cat /a
cat: /a: No such file or directory
# cat /tmp/root-cow/a
a
# echo foo.com > /mnt/etc/mailname
# cat /etc/mailname
glandium.org
# cat /tmp/root-cow/etc/mailname
foo.com
# find /tmp/root-cow
/tmp/root-cow
/tmp/root-cow/etc
/tmp/root-cow/etc/mailname
/tmp/root-cow/a
Keep in mind that it doesn't include submounts, though:
# ls /var
backups cache lib local lock log mail opt run spool tmp www
# ls /mnt/var
This means we'll have to also mount /var, /proc, /dev, and /sys.
That's enough testing for now, and we'll first do some cleanup before going after the real job:
# umount /mnt
# rm -rf /tmp/root-cow
Using the newns utility from my first post on per-process namespaces, let's create a new namespace to keep our testing private, and populate it with the necessary mount points:
# newns
# umount /tmp
# mount -t tmpfs tmpfs /tmp
# mkdir /tmp/root-cow /tmp/var-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt
# mount -t unionfs -o dirs=/tmp/var-cow:/var=ro none /mnt/var
# cd /mnt
# pivot_root . mnt
# mount --move /mnt/proc /proc
# mount --move /mnt/sys /sys
# mount --move /mnt/lib/init/rw /lib/init/rw
# mount --move /mnt/tmp /tmp
# mount --move /mnt/dev /dev
The second and third statements are useful to avoid sharing /tmp with the main namespace, which means the directories we create in the fourth statement won't be visible in /tmp outside this namespace.
The fifth and sixth statements put the union filesystems in place. As my system only has a separate /var filesystem (no /usr, and I don't care about /home here), I only need to setup these two. Feel free to add more union filesystems as necessary.
The pivot_root call allows to switch to the unionfs'ed root: everything under /mnt (/mnt and /mnt/var in our case) will be remounted under /, while what was under / is remounter under /mnt under the new root.
This means, in our case, that we have / and /var as union filesystems, while the old / and /var are respectively in /mnt and /mnt/var.
It also means /dev, /proc, /sys and other filesystems are remounted as /mnt/dev, /mnt/proc, /mnt/sys, etc., which is why we next mount --move all of them to a better place in our namespace.
Once all this setup is done, we are ready to do our php upgrade test. As Etch doesn't have support for neither PID namespaces nor network namespaces, we'll still have some conflicts with the main namespace for TCP port binding and process handling, so we need to be a little careful:
# rm /var/run/apache2.pid
# echo Listen 8080 > /etc/apache2/ports.conf
With these changes, we can now safely start apache2 in this namespace, at the same time the main apache2 runs in the main namespace:
# /etc/init.d/apache2 start
Now, there are actually 2 problems showing up in apache in this setup.
The first one is that displaying static files doesn't work. At all. It appears unionfs doesn't support sendfile(). Which is fine. But apache doesn't check for sendfile()'s error cases and doesn't fallback to a working solution when sendfile() doesn't work. So we have to manually disable it:
# echo EnableSendfile off > /etc/apache2/conf.d/sendfile.conf
The second one is that access to the mysql socket doesn't work properly under the unionfs. I didn't want to investigate further, so I only worked around the issue by forcing to use the socket outside the unionfs:
# mount --bind /mnt/var/run/mysqld /var/run/mysqld
Finally, we can test and validate our php upgrade:
# apt-get install php5-mysql php5-cli php5-snmp libapache2-mod-php5
# /etc/init.d/apache2 restart
# apt-get remove --purge php4-common
Note that installing libapache2-mod-php5 will remove libapache2-mod-php4, and apache2 gets restarted, taking into account this change. But the php modules (php5-mysql and php5-snmp) install is only going to happen after that, and no apache2 restart is triggered then, which leaves a half working php setup...
Also note that the cacti setup in etch supposes php4 is installed, using a IfModule statement against mod_php4.c and none for mod_php5.c, which means a part of its setup doesn't work out of the box (most notably, the DirectoryIndex).
Once all is validated, all we need to do is to stop apache2 and exit from the namespace. The union filesystems and temporary filesystems are then automagically cleaned-up and the namespace disappears as well as all modifications we just did, since all processes using the namespace ended. We are now free to upgrade on the production server, as we know all the side effects.
I'll now be able to upgrade to Lenny without php getting removed.
