• linux
  • debian
  • bsd
  • c
  • python
  • sphinx
  • documentation
  • thinkpad
  • UEFI
  • firmware
  • update
  • git
  • virt
  • filesystem
  • UCS
  • security
  • python,
  • Cloning multiple virtual machines


    I want to clone multiple virtual machines belonging together. They internally communicate among themselves using their static IP addresses. Changing the network configuration per clone is not an option. On the other hand I want to connect to these machines from the outside.

    | +-----------env1------------+                 |
    | | +---vm1.2--+ +---vm1.3--+ |                 |
    | | | | | | +--+              |
    | | +----------+ +----------+ |  |              |
    | +---------------------------+  |              |
    |                                |              |
    | +-----------env2------------+  |              |
    | | +---vm2.2--+ +---vm2.3--+ |  +-------+      |
    | | | | | | +--+ magic +-eth0-+
    | | +----------+ +----------+ |  +--+----+      |
    | +---------------------------+  |              |
    |                                |              |
    | +-----------env3------------+  |              |
    | | ...                       +--+              |
    | +---------------------------+                 |
    Read More

    Debian APT dependency resolver

    Debian’s package manager APT is famous for its inter-package dependency resolving mechanism: Long before rpm based distributions learned how to install dependant packages automatically Debian did do this for many years. You simply can install a high-level package using apt-get install $pkg, which will then automatically resolve all dependencies: Dependant packages are downloaded and install along automatically.

    This works very well when you just use packages from a single consistent source like the stable Debian repository. It mostly also works with multiple repositories, but from time to time the resolver does strange things.

    Here at Univention GmbH we build our own packages. Therefore it is essential for our customers that we get the dependencies right. Today we had the strange behavior, where one of our packages could not be upgraded: APT decided to refuse the package from getting installed. Simplify specifying one additional dependency on the command line made it work.

    So what happened and why did APT refuse the initial command?


    Dependency resolution has some very useful information on how the resolver works:

    APT works in its internal resolver in two stages: First all packages are visited and marked for installation, keep back or removal. Option Debug::pkgDepCache::Marker shows this. This also decides which packages are to be installed to satisfy dependencies, which can be seen by Debug::pkgDepCache::AutoInstall. After this is done, we might be in a situation in which two packages want to be installed, but only one of them can be. It is the job of the pkgProblemResolver to decide which of two packages ‘wins’ and can therefore decide what has to happen. You can see the contenders as well as their fight and the resulting resolution with Debug::pkgProblemResolver.

    Read More

    OCR my PDF

    On a regular basis I have to scan some paper documents. For the scanning I still use xsane (X Scanner Access Now Easy) for that. Afterwards I used GIMP (GUN Image Manipulation Program) to do the post processing:

    • adjust black / white balance to make the background white
    • de-speckle some areas

    This normally reduced the file size by a factor of 10 from 10 MiB per page to roughly 1 MiB per page. While it worked for me it was a lot of work, which took its time.

    Read More

    File System Locking

    Locking files in Linux is tricky - not because it is complex, but complicated due to the many variants.

    First of all you have different kinds:

    • mandatory locks are enforced by the Linux kernel and prevent other processes from opening the file while another process has an exclusive lock. This can easily deadlock the system. Linux supports this on file systems explicitly mounted with option mand.
    • advisory locks require support from all applications: each one has to implement the same mechanism to guarantee proper cooperation.

    For advisory locks you have multiple types in Linux:

    • BSD file lock (<man:flock(2)>)
    • POSIX record lock (<man:fnctl(2)>; the simplified version <man:lockf(3)> only supporting exclusive locks)
    • Open file descriptor (OFD) lock (<man:fcntl(2)>)

    Basically they are incompatible with each other and your applications should agree to use only one.

    Variant Origin NFS range associated fork() auto released
    FLOCK BSD no¹ file open FD inherited last FD closed
    POSIX POSIX yes bytes pid,inode dropped any FDs closed
    OFDLCK Linux yes bytes open FD inherited last FD closed
    Read More

    dpkg --compare-versions

    Q: How are Debian package version strings compared?

    A: This is mandated by Debian Policy and dpkg is considered the single truth of implementation.

    Comparing Debian package version strings is not trivial: many programs implement this themselves and get it wrong for corner cases — me included. Therefor use dpkg –compare-versions or one of its wrappers, for example apt.apt_pkg.version_compare() for Python or debversion for PostgreSQL. Continue reading if you want to understand comparing Debian package version strings yourself, which is important when you increment the version of UCS packages. The format is: [epoch:]upstream-version[-debian-revision]

    Read More

    Python string concatenation

    How fast is string concatenation in Python?

    a + b + c
    "".join((a, b, c))
    "%s%s%s" % (a, b, c)
    "{}{}{}".format(a, b, c)
    Read More

    Faster Debian packages indexing

    You can install Debian packages using dpkg -i $pkg.deb, but this low-level tool does not resolve inter-package dependencies. This is the job of APT, the Advanced Packaging Tool. It usually works on a set of packages, which are shipped in a package repository. This has a Packages file, which lists all binary packages included in the repository. Basically it contains the concatenated package meta data from all packages.

    Read More

    Linux audit system

    The Linux audit system can be used to collect important system events. It is often used for compliance with PCI_DSS 3.1. But it is also useful for debugging certain problems, for example where you have to monitor an unknown number of prcoesses to show a certain behavior. In our case an unknown process kept killing other processes form time to time.

    Read More

    Python XML parsing

    At work we’re using DocBook for our product documentation. We have a tool for spell-checking our documents.

    1. It uses Document Type Definition (DTD) for validation and entity declarations.
    2. The DTD and other referenced files should be cached locally by using an XML catalog.
    3. For mis-spelled words the tool should prints the line and column number.

    Solving this with Python is not easy for obscure reasons.

    Read More

    Python rich comparisons

    Python 3 has switch to Rich Comparisons. With Python 2 is was enough to implement a single __cmp__(self, other) method, now you have to implement

    • __lt__(self, other)
    • __le__(self, other)
    • __eq__(self, other)
    • __ne__(self, other)
    • __ge__(self, other)
    • __gt__(self, other)
    • __hash__(self)

    @functools.total_ordering helps with this, but this is still painful.

    Read More

    Prevent eCryptfs from asking for passphrase

    For historical reasons I have been using eCryptfs, a file system layer for encrypted files. It got removed from Debian Buster, but I’m still using it.

    For transparent usage it installs its own PAM module: When you log in your password can be used to automatically decrypt your files. You can also use a different passphrase to improve security even more.

    But this shows an annoying behavior, as you also get asked for that additional passphrase when you use sudo or other tools.

    I (temporarily) fixed this by changing my /etc/pam.d/common-auth to use pam_succeed_if like this:

    auth    [default=1 success=ignore]      pam_succeed_if.so service notin sudo:polkit-1
    auth    optional        pam_ecryptfs.so unwrap

    This skips the call to pam_ecryptfs if the service is either sudo or PolicyKit-1, which is used by the update service.

    Read More

    Speedup Debian package building

    For my employee Univention I want to setup a Continuous Integration system to build Debian packages. We have been using our own build-system called Repo-NG based on pbuilder, which show their age. I tried several tricks to improve the build speed as we have to build many packages and many of them multiple times.

    Read More

    btrfs snapshots for send and receive

    I’m using MythTV for watching TV and videos. For that I have a separate HP micro server which 4 HDs. That system is quiet old and low-powered. Because of that I rip my DVDs on my new PC and copy over the files.

    Previously I’ve used rsync to synchronize the films over to the MythTV system. Last week I made a mistake and destoryed several files. I only noticed my mistake after I had synchronized the files, so my “backup” was gone as well.


    Therefore I switched both systems to use btrfs which allows to create snapshots.

    Read More

    Bareos Shared Secrets

    Bareos is a backup software, which was forked from Bacula years ago. It consists of several services:

    • Bareos Director (D) @ TCP:9101: central scheduler to coordinate all activities.
    • Bareos File Daemon (FD) @ TCP:9102: service running on each client to backup (and restore) the client.
    • Bareos Storage Daemon (SD) @ TCP:9103: storage system to host the backed-up data.
    • Bareos Catalog (C): database with data of all backed-up files.
    • Bareos Console / WebUI: textual or graphical interface to the Director.
    • Bareos Tray-Monitor (TM): Desktop application for monitoring other services.
    Read More

    Linux Kernel Keyring Quota exceeded

    Nach einer kurzen Pause kam ich heute wieder zurück zu meinem Notebook und konnte mich nicht mehr anmelden. Nach einigem Suchen bin ich in journalctl -u sssd über folgende Fehlermeldung gestolpert:

    [sssd[krb5_child[18654]: Disk quota exceeded
    Read More

    Minimal Debian images

    At work I need minimal Docker images. debootstrap is Debians default way to create chroot environments. By default they include all required and essential packages. For a hardware system this is okay, but too much for a container image.

    Read More

    Ceph-Jewel RBD libvirt storage pool

    For some development work on an Univention Corporate Server 4.4, which is based on Debian Stretch, I needed a Ceph cluster based on the Jewel release. Most of the tutorials were based on newer Ceph releases (Luminous, Mimic) or were using ceph-deploy, which is not part of Debian and must be installed separately.

    Therefor I did a manual installation, using the low-level tools. In contrast to Ceph Storage with UCS I do not want to use CephFS, but use Rados directly.

    This is a test setup and not appropriate for production:

    • I’m only running one monitor, which is a single point of failure.
    • I disabled replication on purpose.
    • all nodes have a single big ext4 file system, so no partitioning, no fast journal disks, and issues with extended attributes.
    Read More

    Bootable Debian Bookworm on USB stick

    Debian 10 Buster was release at the beginning of July 2019. Debian 11 Bullseye was release at the beginning of July 2021. Debian 12 Bookworm was release at the beginning of July 2023. But how to make a bootable USB stick from it?

    Read More

    VirtIO Memory Ballooning

    VirtIO provides Memory Ballooning: the host system can reclaim memory from virtual machines (VM) by telling them to give back part of their memory to the host system. This is achieved by inflating the memory balloon inside the VM, which reduced the memory available to other tasks inside the VM. Which memory pages are given back is the decision of the guest operating system (OS): It just tells the host OS which pages it does no longer need and will no longer access. The host OS then un-maps those pages from the guests and marks them as unavailable for the guest VM. The host system can then use them for other tasks like starting even more VMs or other processes.

    If later on the VM need more free memory itself, the host can later on return pages to the guest and shrink the holes. This allows to dynamically adjust the memory available to each VM even while the VMs keep running.

    Read More

    Reproducible Build

    Debian had started to make their build reproducible: Two builds of the same source package should produce bit identical binary packages. This allows anybody to verify that nobody tempered with the build system.

    Read More


    CPUID is an assembler instruction to identify Intel compatible CPUs. Calling that instruction with register EAX set to 1 returns information about the CPU model in register EBX.

    Read More

    Lenovo ThinkPad L470 Firmware update with Linux

    My company notebook (A Lenovo ThinkPad L470) sometimes crashed when I put it into the docking station: It turn back on, the external monitor turns on, but after that I only see a black screen with the mouse cursor. Today I had enough and performed the pending firmware update, which also includes the Intel CPU microcode updates.

    Read More

    Secure Boot with Linux


    Unwanted binaries like viruses should be prevented from loading. This is known as Secure-Boot. The (U)EFI firmware only loads binaries signed by the “Platform key” (PK) certificates. The PK is pre-installed by the manufacturer. Probably 9x% come with Microsoft Windows pre-installed. Therefor most PCs come with Microsoft key pre-installed. For Qemu/KVM there is “OVMF”: It is based on the EDK2 (EFI Development KIT). It is developed by the “TianoCore” community. It has not keys pre-installed.

    Read More