Q: How do you write a debug helper function passing through printf() like data?

A: Use a combination of macros and variadic functions combined with the power of GCC extensions.

Several years ago I did setup OwnCloud. I managed users and groups in OpenLDAP, as I had another use-cases, where I needed that information for authentication.

Several years later I switched to NextCloud and got rid of that other use-case. Managing users and groups have become a pain in LDAP: I have to use CLI tools instead of the GUI. While I can do it, others don’t like it. So I would like to uninstall OpenLDAP.

Nicols ask the same question and developed a solution: Import LDAP users & get rid of LDAP It is a bit outdated and contains some errors. So here is my step-by-step guide.

Q: How do I recursively find all shell scripts in my current working directory?

A: Search for the hash-bang line:

$ git -c grep.fallbackToNoIndex=yes grep -lPe '\A#!\s*/bin/([bd]?a)?sh\b'

I work with GitLab, where I often have to create Merge Requests. You can do this using the browser based graphical user interface, or from the command line when doing a git push: You can specify several push options for merge requests via the argument -o:

• merge_request.create: Create a new merge request for the pushed branch.
• merge_request.target=<branch_name>: Set the target of the merge request to a particular branch.
• merge_request.merge_when_pipeline_succeeds: Set the merge request to merge when its pipeline succeeds.
• merge_request.remove_source_branch: Set the merge request to remove the source branch when it’s merged.

There are many more.

Some time ago I received a new Lenovo P14s notebook. My previous L470 worked very well for a long time, so I only switch to the new model beginning this year. Since then I experienced regular stalls: Mostly during our daily video conferences the laptop locked up for some minutes; only after 1-2 minutes I was able to resume my work, but found some process(es) gone.

Problem

I want to clone multiple virtual machines belonging together. They internally communicate among themselves using their static IP addresses. Changing the network configuration per clone is not an option. On the other hand I want to connect to these machines from the outside.

┌──────────────────────host─────────────────────┐
│ ┌───────────env1────────────┐                 │
│ │ ┌───vm1.2──┐ ┌───vm1.3──┐ │                 │
│ │ │ 10.0.0.2 │ │ 10.0.0.3 │ ├──┐              │
│ │ └──────────┘ └──────────┘ │  │              │
│ └───────────────────────────┘  │              │
│                                │              │
│ ┌───────────env2────────────┐  │              │
│ │ ┌───vm2.2──┐ ┌───vm2.3──┐ │  ├───────┐      │
│ │ │ 10.0.0.2 │ │ 10.0.0.3 │ ├──┤ magic ├─eth0─┼─
│ │ └──────────┘ └──────────┘ │  ├───────┘      │
│ └───────────────────────────┘  │              │
│                                │              │
│ ┌───────────env3────────────┐  │              │
│ │ ...                       ├──┘              │
│ └───────────────────────────┘                 │
└───────────────────────────────────────────────┘

In my previous blog post Python rich comparison I looked at simplifying the comparison of objects.

Using NotImplemented in boolean context has been deprecated since Python 3.9: As bool(NotImplemented) is True this resulted in many wrong implementations, including mine.

So how do you correctly implement rich comparison?

Python wheels are a distribution format for Python packages defined by PEP-427. As long as your package is simple and just consists of a bunch of Python files, creating and shipping a source only distribution is probably fine. But as soon your Python package build process takes time or needs (many) other dependencies to build — or even worse — it needs some (C-)compiler and libraries plus their development headers, those packages become a pain.

Debian’s package manager APT is famous for its inter-package dependency resolving mechanism: Long before rpm based distributions learned how to install dependant packages automatically Debian did do this for many years. You simply can install a high-level package using apt-get install $pkg, which will then automatically resolve all dependencies: Dependant packages are downloaded and install along automatically.

This works very well when you just use packages from a single consistent source like the stable Debian repository. It mostly also works with multiple repositories, but from time to time the resolver does strange things.

Here at Univention GmbH we build our own packages. Therefore it is essential for our customers that we get the dependencies right. Today we had the strange behavior, where one of our packages could not be upgraded: APT decided to refuse the package from getting installed. Simplify specifying one additional dependency on the command line made it work.

So what happened and why did APT refuse the initial command?

Resolver

Dependency resolution has some very useful information on how the resolver works:

APT works in its internal resolver in two stages: First all packages are visited and marked for installation, keep back or removal. Option Debug::pkgDepCache::Marker shows this. This also decides which packages are to be installed to satisfy dependencies, which can be seen by Debug::pkgDepCache::AutoInstall. After this is done, we might be in a situation in which two packages want to be installed, but only one of them can be. It is the job of the pkgProblemResolver to decide which of two packages ‘wins’ and can therefore decide what has to happen. You can see the contenders as well as their fight and the resulting resolution with Debug::pkgProblemResolver.

On a regular basis I have to scan some paper documents. For the scanning I still use xsane (X Scanner Access Now Easy) for that. Afterwards I used GIMP (GUN Image Manipulation Program) to do the post processing:

• adjust black / white balance to make the background white
• de-speckle some areas

This normally reduced the file size by a factor of 10 from 10 MiB per page to roughly 1 MiB per page. While it worked for me it was a lot of work, which took its time.

Locking files in Linux is tricky - not because it is complex, but complicated due to the many variants.

First of all you have different kinds:

• mandatory locks are enforced by the Linux kernel and prevent other processes from opening the file while another process has an exclusive lock. This can easily deadlock the system. Linux supports this on file systems explicitly mounted with option mand.
• advisory locks require support from all applications: each one has to implement the same mechanism to guarantee proper cooperation.

For advisory locks you have multiple types in Linux:

• BSD file lock (<man:flock(2)>)
• POSIX record lock (<man:fnctl(2)>; the simplified version <man:lockf(3)> only supporting exclusive locks)
• Open file descriptor (OFD) lock (<man:fcntl(2)>)

Basically they are incompatible with each other and your applications should agree to use only one.

In the early days of K&R C programming the type for functions parameters was optional and declared separately:

void old(a, b, c)
 int a;
 bool b;
 char *c;
{
 printf("a=%i b=%d c=%p\n", a, b, c);
}

Q: How to convert from legacy MBR to GPT required for UEFI?

A: Follow ServerFault: How do I convert my linux disk from MBR to GPT with UEFI

• Creating the “BIOS boot partition” is only needed if you want to do the conversion in small steps, e.g. convert to GPT first and still use grub-pc for booting. You can skip this step if you convert to GPT and grub-efi-amd64-bin at the same time.

During Debian package installation, upgrade, downgrade and remove the so called Debian Package maintainer scripts are called before and after certain actions:

argparse has superseded optparse, which is deprecated since Python 3.2 and might get removed in the future. Many of our scripts have already been migrated, but argparse and the migration has several pitfalls. This was already mentioned in one of my previous posts Python: optparse vs. argparse.

Q: How are Debian package version strings compared?

A: This is mandated by Debian Policy and dpkg is considered the single truth of implementation.

Comparing Debian package version strings is not trivial: many programs implement this themselves and get it wrong for corner cases — me included. Therefor use dpkg –compare-versions or one of its wrappers, for example apt.apt_pkg.version_compare() for Python or debversion for PostgreSQL. Continue reading if you want to understand comparing Debian package version strings yourself, which is important when you increment the version of UCS packages. The format is: [epoch:]upstream-version[-debian-revision]

How fast is string concatenation in Python?

a + b + c
"".join((a, b, c))
"%s%s%s" % (a, b, c)
"{}{}{}".format(a, b, c)
f"{a}{b}{c}"
c="";c+=x;c+=y;c+=z;c

You can install Debian packages using dpkg -i $pkg.deb, but this low-level tool does not resolve inter-package dependencies. This is the job of APT, the Advanced Packaging Tool. It usually works on a set of packages, which are shipped in a package repository. This has a Packages file, which lists all binary packages included in the repository. Basically it contains the concatenated package meta data from all packages.

The Linux audit system can be used to collect important system events. It is often used for compliance with PCI DSS 3.1. But it is also useful for debugging certain problems, for example where you have to monitor an unknown number of processes to show a certain behavior. In our case an unknown process kept killing other processes form time to time.

At work we’re using DocBook for our product documentation. We have a tool for spell-checking our documents.

1. It uses Document Type Definition (DTD) for validation and entity declarations.
2. The DTD and other referenced files should be cached locally by using an XML catalog.
3. For misspelled words the tool should prints the line and column number.

Solving this with Python is not easy for obscure reasons.

Continuing my GitLab and Kubernetes (k8s) odyssey from k8s @ Debian I’ve learned two things:

1. DNS problems with Alpine Linux
2. SSL certificate hell

Recently I’ve been reeding books:

• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win
• The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data
• The DevOps Handbook

Python 3 has switch to Rich Comparisons. With Python 2 is was enough to implement a single __cmp__(self, other) method, now you have to implement

• __lt__(self, other)
• __le__(self, other)
• __eq__(self, other)
• __ne__(self, other)
• __ge__(self, other)
• __gt__(self, other)
• __hash__(self)

@functools.total_ordering helps with this, but this is still painful.

For historical reasons I have been using eCryptfs, a file system layer for encrypted files. It got removed from Debian Buster, but I’m still using it.

For transparent usage it installs its own PAM module: When you log in your password can be used to automatically decrypt your files. You can also use a different passphrase to improve security even more.

But this shows an annoying behavior, as you also get asked for that additional passphrase when you use sudo or other tools.

I (temporarily) fixed this by changing my /etc/pam.d/common-auth to use pam_succeed_if like this:

auth    [default=1 success=ignore]      pam_succeed_if.so service notin sudo:polkit-1
auth    optional        pam_ecryptfs.so unwrap

This skips the call to pam_ecryptfs if the service is either sudo or PolicyKit-1, which is used by the update service.

Working for libvirt I had to add the Developer Certificate of Origin to several previous commits, where I forgot to directly use git commit --signoff. StackOverflow has that question, but it started missing -s for --signoff with -S for --gpg-sign.

Q: Wie unterscheide in Upgrades von Neuinstallationen?

A: dpkg --compare-versions "$2" lt-nl "…"

I had to build a web camera for watching some newly hatched birds. From previous experiments I already had a spare Raspberry Pi wit an attached camera. Due to bandwidth limitations in the wireless network I wanted to host the resulting media on one of my public servers. I’m using ffmpeg with HLS streaming as this works in most browsers out-of-the-box.

Debians dpkg-buildpackage has the annoying feature, that the build artifacts are placed in the parent directory. Bug 657491 requested that feature in 2012, but there still is no such option. While working on speeding up the build process I investigated, how that could be fixed or at least be worked around.

For my employee Univention I want to setup a Continuous Integration system to build Debian packages. We have been using our own build-system called Repo-NG based on pbuilder, which show their age. I tried several tricks to improve the build speed as we have to build many packages and many of them multiple times.

I’m using MythTV for watching TV and videos. For that I have a separate HP micro server which 4 HDs. That system is quiet old and low-powered. Because of that I rip my DVDs on my new PC and copy over the files.

Previously I’ve used rsync to synchronize the films over to the MythTV system. Last week I made a mistake and destroyed several files. I only noticed my mistake after I had synchronized the files, so my “backup” was gone as well.

Uups.

Therefore I switched both systems to use btrfs which allows to create snapshots.

Bareos is a backup software, which was forked from Bacula years ago. It consists of several services:

• Bareos Director (D) @ TCP:9101: central scheduler to coordinate all activities.
• Bareos File Daemon (FD) @ TCP:9102: service running on each client to backup (and restore) the client.
• Bareos Storage Daemon (SD) @ TCP:9103: storage system to host the backed-up data.
• Bareos Catalog (C): database with data of all backed-up files.
• Bareos Console / WebUI: textual or graphical interface to the Director.
• Bareos Tray-Monitor (TM): Desktop application for monitoring other services.

Nach einer kurzen Pause kam ich heute wieder zurück zu meinem Notebook und konnte mich nicht mehr anmelden. Nach einigem Suchen bin ich in journalctl -u sssd über folgende Fehlermeldung gestolpert:

[sssd[krb5_child[18654]: Disk quota exceeded

At work I need minimal Docker images. debootstrap is Debians default way to create chroot environments. By default they include all required and essential packages. For a hardware system this is okay, but too much for a container image.

For some development work on an Univention Corporate Server 4.4, which is based on Debian Stretch, I needed a Ceph cluster based on the Jewel release. Most of the tutorials were based on newer Ceph releases (Luminous, Mimic) or were using ceph-deploy, which is not part of Debian and must be installed separately.

Therefor I did a manual installation, using the low-level tools. In contrast to Ceph Storage with UCS I do not want to use CephFS, but use Rados directly.

This is a test setup and not appropriate for production:

• I’m only running one monitor, which is a single point of failure.
• I disabled replication on purpose.
• all nodes have a single big ext4 file system, so no partitioning, no fast journal disks, and issues with extended attributes.

Q: How do you find the process listening on an UNIX domain socket?

This is some WIP. Setup GitLab in Kubernetes (k8s) test cluster using the helm chart:

Debian 10 Buster was release at the beginning of July 2019. Debian 11 Bullseye was release at the beginning of July 2021. Debian 12 Bookworm was release at the beginning of July 2023. But how to make a bootable USB stick from it?

VirtIO provides Memory Ballooning: the host system can reclaim memory from virtual machines (VM) by telling them to give back part of their memory to the host system. This is achieved by inflating the memory balloon inside the VM, which reduced the memory available to other tasks inside the VM. Which memory pages are given back is the decision of the guest operating system (OS): It just tells the host OS which pages it does no longer need and will no longer access. The host OS then un-maps those pages from the guests and marks them as unavailable for the guest VM. The host system can then use them for other tasks like starting even more VMs or other processes.

If later on the VM need more free memory itself, the host can later on return pages to the guest and shrink the holes. This allows to dynamically adjust the memory available to each VM even while the VMs keep running.

Debian had started to make their build reproducible: Two builds of the same source package should produce bit identical binary packages. This allows anybody to verify that nobody tempered with the build system.

git-filter-branch can be used to rewrite the history of one or more branches. As a Debian Developer working for Univention GmbH I often have to work with Debian packages. Here are some more examples from my daily work.

If you run the following shell script, it will terminate:

seq | head -n 1

But why?

CPUID is an assembler instruction to identify Intel compatible CPUs. Calling that instruction with register EAX set to 1 returns information about the CPU model in register EBX.

My company notebook (A Lenovo ThinkPad L470) sometimes crashed when I put it into the docking station: It turn back on, the external monitor turns on, but after that I only see a black screen with the mouse cursor. Today I had enough and performed the pending firmware update, which also includes the Intel CPU microcode updates.

Scan directory univention for Python modules and generate .rst files in directory sphinx:

In the beginning, there was void.

As a C programmer you probably know strcpy(), but hopefully strncpy(), too.

But do you also know strlcpy() (from BSD) and strscpy() (from the Linux kernel)?

And do you know the suitable differences?

Summary

Calling strXcpy(dst, "text", sizeof dst) returns … and afterwards dst has … with

• ¶ the NUL-byte
• · unmodified
• 💥 Crash

Sometimes you add a temporary APT source to /etc/apt/sources.list, install same packages and remove the repository again. Or you install some .deb file directly, which you copied by hand to your environment.

RFC 2368 specified the mailto: syntax. It was superseded by RFC 6068, which added UTF-8 support. It can be used to launch your email client by clicking on some URL.

Some of my notes on using FFmpeg.

Why

Unwanted binaries like viruses should be prevented from loading. This is known as Secure-Boot. The (U)EFI firmware only loads binaries signed by the “Platform key” (PK) certificates. The PK is pre-installed by the manufacturer. Probably 9x% come with Microsoft Windows pre-installed. Therefor most PCs come with Microsoft key pre-installed. For QEMU/KVM there is “OVMF”: It is based on the EDK2 (EFI Development KIT). It is developed by the “TianoCore” community. It has not keys pre-installed.

I had to search man 1 find too often for printf:

Kleine Denksportaufgabe für den Morgen:

Was gibt folgendes Skript aus, d.h. was wird durch das set -e als Fehler gewertet und führt zum Abbruch des Skripts:

#!/bin/sh
set -e
! false ; echo 0
! true ; echo 1
true && true ; echo 2
true || true ; echo 3
false && true ; echo 4
false || true ; echo 5
false && false ; echo 6
true || false ; echo 7
true && false ; echo 8
false || false ; echo 9

f () { false; echo 10; }
if f; then echo 11; else echo 12; fi

Mit Bug 36044 hatten wird das Problem, das eine geöffnete TCP-Verbindung zu unserem Update-Server steckengeblieben ist und damit unsere Tests solange blockiert hat, bis wir sie von Hand abgebrochen haben.

Aus der Kategorie “Nützliche Tools” für das Schreiben von (UCS-)Tests:

Teil 2 von „Schlaflos in Oldenburg“:

from subprocess import Popen, PIPE
producer = Popen(("producer",), stdout=PIPE)
consumer = Popen(("consumer",), stdin=producer.stdout)
ret = producer.wait()

Wo ist das Problem?

Ich könnte diese Folge auch „Schlaflos in Oldenburg“ nennen, aber hier ein Beispiel, wie man es nicht macht (in Auszügen):

Wenn man das Zeichen-Encoding richtig machen will, empfiehlt es sich, intern mit Unicode zu arbeiten und bei jedem Zugriff auf das Dateisystem die Daten zwischen dem internen Unicode-Format und der externen Kodierung zu konvertieren. Leider ist Python-2 aus Kompatibilitätsgründen dazu gezwungen, sein altes kaputtes Verhalten beizubehalten. Besser wird es erst mit Python 3, wo dann alle Zeichenketten standardmäßig Unicode verwenden und man explizit sagen muß, wenn man ein Byte-Array haben möchten. Kaputt deswegen, weil:

Das devsync-Programm aus dem toolshed ermöglicht es ja, Dateien vom lokalen System in eine VM zu synchronisieren, um darin dann den Bauvorgang anzustoßen.
Mit QEmu/KVM auf dem lokalen System geht es übrigens noch ein bisschn besser, denn per 9p (vom Betriebssystem Plan 9) kann man ein lokales Verzeichnis auch direkt innerhalb einer lokalen QEmu-VM einbinden, je nach Bedarf nur-lesend oder auch durchlässig in beide Richtungen.

Der Ein oder Andere wird es vielleicht mitbekommen haben, aber einer unserer Kunden hat Ende Januar die Gefährlichkeit eines rm -rf / als Benutzer root erfahren (für die nicht-Techniker: Lösche rekursiv alle Dateien und Verzeichnisse, angefangen beim Wurzelverzeichnis und ohne weiteres nachfragen). Zum Glück hatte der Kunde ein 18h altes Backup, aber trotzdem ist viel Arbeit über den Jordan gegangen, weil unter anderem auch die ganzen virtuellen Maschinen per NFS unter /nfs/ gemounted waren… die Betonung liegt hier auf „waren“ 🙁 Wäre es an der Stelle nur ein rm gewesen, wäre die Sache vermutlich auch noch gut ausgegangen, denn ein rm -rf / kann man gefahrlos einfach mal so eingeben:

Wenn KVM-Instanzen das latest-ISO-Image einbinden kommt es immer wieder vor, daß nach dem Neubau der Image-Datei die VM Lesefehler meldet:

Auf der Suche nach $1 gleichen Zeichen $2 hatte ich ursprünglich folgenden Code:

ruler () {  # count char
    local i
    for ((i=0;i<$1;i++))
    do
        echo -n "${2:-=}"
    done
}

dpkg behandelt alle Dateien unterhalb von /etc/ gesondert, da es sich dabei um sog. conffiles handelt: “Änderungen daran durch den Benutzer müssen laut Debian-Policy selbst bei einem Paket-Upgarde erhalten bleiben.” Diese werden auch nicht bei einem normalen remove entfernt, sondert erst bei einem purge. Dazu speichert dpkg für jede Konfigurationsdatei in /var/lib/dpkg/status die md5-Summe der Originaldatei, um geänderte Dateien zu erkennen. (Diese lassen sich durch dpkg-query -W -f '${Conffiles}\n' "$pkg_name" auslesen).

Leider ist das Erstellen, Zurückspielen und Löschen von Sicherungspunkte sowohl in unserer Testumgebung (und leider auch bei Kunde) mit qemu-0.14.0 extrem langsam: Standardmäßig verwendet KVM als Cache-Strategie writetrough, was dafür sorgt, das Änderungen sofort auf die Festplatte zurückgeschrieben werden, um Datenverlust bei Stromausfall vorzubeugen. Bei den Dateioperationen für Sicherungspunkte müssen an sehr vielen Stellen Änderungen gemacht werden, was dann zu der schlechten Performance führt, insbesondere wenn auch noch NFS verwendet wird.

Das Python subprocess Modul hat so seine Tücken:

Mit Virtualisierung trifft man öfters auf mehrere Gigabyte große Image-Dateien, die von Zeit zu Zeit auch mal kopiert werden müssen. Klassischerweise passiert das mit cp oder dd bs=1G. Beide Varianten lesen die Daten häppchenweise per read() in den Hauptspeicher, um ihn anschließend per write() in eine neue Datei zu schreiben. Die dd-Varianta hat hier den Vorteil, daß man durch Angab der Blockgröße größere Datenblöcke am Stück einlesen kann, was vor allem die Anzahl der Lese-/Schreibkopfpositionierungen der Festplatte reduziert, was zu erheblichen Geschwindigkeitsvorteilen führen kann. cp verwendet normalerweise 32 KiB-Blöcke, shutil.copy2() aus Python nur 16 KiB-Blöcke (und kopiert nicht alle Berechtigungen!)

Zum Testen des Univention-Updaters wird dieser oft auf einem alten UCS-System installiert und dann damit ein Update auf die aktuellste Version durchgeführt. Dabei mischt man Pakete aus dem offiziellen UCS-Depot mit neueren Paketen von omar. Um sicherzustellen, daß man nur den neueren Updater und nicht auch weitere Pakete von omar verwendet, kann man apt-Pinnging verwenden:

Bei KVM wird fast immer das Qcow2-Dateiformat für die Speicherung des Festplattenimmages verwendet. Dieses Format belegt zunächst wenig Speicher, wächst aber bei steigender Nutzung durch das Gast-Betriebssystem bis zur angegebenen Höchstgrenze an. Durch Sicherungspunkte kann so eine Datei aber auch deutlich größer sein als die nominal vorgegebene Größe. Dies kann u.a. dazu führen, das im Betrieb das Dateisystem des Host-Systems voll läuft und die Qcow2-Datei nicht weiter anwachsen kann. In diesem Fall pausieren die betroffenen KVM-Instanzen, so daß es hier nicht zu Datenverlust kommt, solange das nicht von Hand per /domain/devices/disk/driver/@error_policy="stop|ignore|enospace" geändert wurde. Nachdem man wieder freien Speicherplatz geschaffen hat, muß die VM per Kommandozeilentool fortgesetzt werden: sudo virsh qemu-monitor-command "$VM" cont.

Nach dem Pizza-Abend-Vortrag von Jan-Christoph ist irgendwann mal das set hidden in meine ~/.vimrc gewandert, mit dem Dateien durch ein :q nicht sofort geschlossen werden, sondern in den Hintergrund wandern, so daß man ggf. schnell dort weitermachen kann, wenn man die Datei dann doch nochmal öffnen und weiter bearbeiten will. Nachteilig war das allerdings im Zusammenhang mit VCS-Plugin, weil sich nach einem VCSVimDiff dieser Modus nicht mehr ordentlich beendet hat: Die SVN-Datei blieb auch als Puffer geöffnet, was dazu geführt hat, das im Hintergrund diese weiterhin mit der Arbeitsdatei verglichen wurde. Nach langer Suche habe ich dann :bdelete entdeckt, mit dem man einen Puffer selbst bei gesetzten hidden schließen kann.

Sönke suchte eine einfache Möglichkeit, alle veralteten Kernel von einem System zu löschen. Das sollte folgender Einzeiler erledigen: