pkgsrc (2/6)
This post is an automatic translation from French. You can read the original version here.
And here we are again for another round of note-taking on Imil’s streams! In this video, we continue discovering the pkgsrc system and the life of a package maintainer.
Previously on Imil TV…
Let’s start with a quick recap of what we saw last time. Interested in a piece of software available on gitlab, truefalse v1.0, we decided to package it with pkgsrc…
Pkgsrc is a package management system made in NetBSD and based on bmake. It works on many other platforms, among which we can note Mac OSX and even Windows! The commands presented here are actually tested on Gentoo by yours truly :)
We started our adventure in the world of package maintainers by using a tool called url2pkg, which turns a URL into a package:
$ mkdir ~/pkgsrc/wip/truefalse
$ cd ~/pkgsrc/wip/truefalse
$ url2pkg https://gitlab.com/iMil/truefalse/-/archive/v1.0/truefalse-v1.0.tar.gzWe get a Makefile, which we tweak a bit:
PROGNAME= truefalse
DISTNAME= truefalse-v1.0
PKGNAME= ${DISTNAME:S,-v,-,}
CATEGORIES= misc
MASTER_SITES= https://gitlab.com/iMil/truefalse/-/archive/v1.0/
MAINTAINER= truefalse@rancune.org
HOMEPAGE= https://gitlab.com/iMil/truefalse/-/archive/v1.0/
COMMENT= First attempt at making a package
LICENSE= original-bsd
INSTALLATION_DIRS= bin
do-install:
${INSTALL_PROGRAM} ${WRKSRC}/${PROGNAME} ${DESTDIR}/${PREFIX}/bin/${PROGNAME}
.include "../../mk/bsd.pkg.mk"Normally, to compile a package, you use the ./configure command, which generates a Makefile with an install target. Then you simply run:
make installBut in the project we are packaging, the Makefile is not very sophisticated. It is a very simple bmake Makefile that does not have an installation target.
This is why we overrode the do-install target in our package. Hence these lines in particular:
do-install:
${INSTALL_PROGRAM} ${WRKSRC}/${PROGNAME} ${DESTDIR}/${PREFIX}/bin/${PROGNAME}At this point, a quick reminder may be necessary regarding the DESTDIR and PREFIX variables. You are probably familiar with PREFIX if you have already worked with autotools and the famous “configure” script. Indeed, by default, this configure script uses a prefix of “/usr/local” for installations. This allows choosing the base installation path: binaries go to /usr/local/bin, libraries to /usr/local/lib and so on… If you want to deploy the software elsewhere, in /usr for example, just modify the PREFIX variable.
Pkgsrc introduces a second variable, DESTDIR, which allows us to “simulate” an installation in a directory using the stage-install target. If you type:
bmake stage-installThe installation will not happen in the usual way but locally, in the work subdirectory. This is very handy for verifying the installation destinations of our package.
When we type “bmake install”, DESTDIR equals “/”.
When we type “bmake stage-install”, DESTDIR equals “/home/rancune/pkgsrc/wip/truefalse/work”.
Simple as that!
pkgsrc and its targets
A quick word about the different targets offered by pkgsrc. There are quite a few and they are rather handy! The interested reader can visit:
https://wiki.netbsd.org/pkgsrc/intro_to_packaging/
Here is a small selection:
- bmake stage-install
Allows a simulated installation in a directory local to the package, as seen above.
- bmake print-PLIST
Lists the package files installed on the filesystem
- bmake clean
Deletes the source files from the working directory, for a clean fresh start!
- bmake distclean
Deletes the downloaded source archive
- bmake fetch
Fetches the file and verifies if its checksum matches the recorded one
- bmake extract
Extracts (decompresses) the source files from the archive and places them in the working directory
- bmake patch
Applies the (package) patches to the sources
- bmake show-depends
Allows you to view the package dependencies, if any:
$ cd /home/rancune/pkgsrc/x11/gnome-terminal
$ bmake show-depends
dconf>=0.36.0nb6:../../devel/dconf
glib2>=2.70.2:../../devel/glib2
libuuid>=2.18:../../devel/libuuid
hicolor-icon-theme>=0.9nb1:../../graphics/hicolor-icon-theme
desktop-file-utils>=0.10nb1:../../sysutils/desktop-file-utils
gsettings-desktop-schemas>=3.4.2:../../sysutils/gsettings-desktop-schemas
vte3>=0.60.3nb12:../../x11/vte3
- bmake show-options
Packages can sometimes have options, meaning features that you can choose to compile or not when installing said package. To view these options, proceed as follows:
$ cd /home/rancune/pkgsrc/multimedia/vlc
$ bmake show-options
Any of the following general options may be selected:
alsa Enable ALSA support.
avahi Enable DNS service discovery and multicast DNS support.
dbus Enable dbus (desktop bus) support.
debug Enable debugging facilities in the package.
dts Enable DTS Coherent Acoustics support.
jack Enable support for the JACK audio server.
lirc Enable LIRC receiving and sending IR signals support.
pulseaudio Enable support for the PulseAudio sound server.
qt5 Enable support for QT5.
vaapi Enable support for VAAPI.
vdpau Enable support for VDPAU.
vlc-skins Enable skins support in VLC.
x11 Enable X11 support.
These options are enabled by default:
alsa dbus lirc qt5 vaapi vdpau x11
These options are currently enabled:
alsa dbus lirc qt5 vaapi vdpau x11
You can select which build options to use by setting PKG_DEFAULT_OPTIONS
or PKG_OPTIONS.vlc.As indicated by the displayed message, option selection is done through the PKG_DEFAULT_OPTIONS variable for general choices, and through the PKG_OPTIONS.<package> variable for a specific package.
This is done in the mk.conf file, which contains the pkgsrc configuration.
- make show-var VARNAME=<variable>
Allows you to query the value of a variable used in pkgsrc.
$ bmake -V INSTALL_PROGRAM
${INSTALL} ${COPY} ${_STRIPFLAG_INSTALL} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE}
$ bmake show-var VARNAME=INSTALL_PROGRAM
/usr/bin/install -c -s -o rancune -g rancune -m 755Personal opinion: I am increasingly convinced by pkgsrc. I find the system remarkably efficient, don’t you?
Let’s now return to our initiation in the way of the packager :)
Writing a patch:
We now return to our package and our project truefalse v1.0. We recompile the project to get back to the same point as last week:
$ cd /home/rancune/pkgsrc/wip/truefalse
$ bmake install cleanWhen we had tested this program, we noticed it did not work correctly: launched without arguments, the program returns false!
And for good reason:
#include <stdlib.h>
int
main(int argc, char *argv[])
{
if (argc < 1)
return EXIT_SUCCESS;
return EXIT_FAILURE;
}You can see the bug: the test should be (argc<2) and not (argc<1)!
As packagers, we decide to fix the bug and propose an upstream patch (to the developers of the original project) correcting this little error!
We start by extracting the truefalse sources:
$ bmake extract
=> Bootstrap dependency digest>=20211023: found digest-20220214
=> Checksum BLAKE2s OK for truefalse-v1.0.tar.gz
=> Checksum SHA512 OK for truefalse-v1.0.tar.gz
===> Installing dependencies for truefalse-1.0
=> Tool dependency cwrappers>=20150314: found cwrappers-20220403
===> Checking for vulnerabilities in truefalse-1.0
===> Overriding tools for truefalse-1.0
===> Extracting for truefalse-1.0In the meta-package pkg-developer that we installed last time, there is an awesome tool: pkgvi!
$ pkgvi work/truefalse-v1.0/truefalse.c
pkgvi: File was modified. For a diff, type:
pkgdiff "work/truefalse-v1.0/truefalse.c"
$ ls work/truefalse-v1.0/
total 16
-rw-r--r-- 1 rancune rancune 46 22 avril 16:33 Makefile
-rw-r--r-- 1 rancune rancune 81 22 avril 16:33 README.md
-rw-r--r-- 1 rancune rancune 120 1 mai 12:24 truefalse.c
-rw-r--r-- 1 rancune rancune 120 22 avril 16:33 truefalse.c.origpkgvi opens a vim window to edit the code, and automatically creates the truefalse.c.orig containing the code before modification. Isn’t that classy?
To create the patch, we once again call upon pkgsrc and the mkpatches command:
$ mkpatches
$ ls
total 28
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 '=20150314:'
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 '=20211023:'
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Bootstrap
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Checking
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Checksum
-rw-r--r-- 1 rancune rancune 81 25 avril 12:15 DESCR
-rw-r--r-- 1 rancune rancune 312 25 avril 12:10 distinfo
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Extracting
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Installing
-rw-r--r-- 1 rancune rancune 451 25 avril 12:37 Makefile
-rw-r--r-- 1 rancune rancune 425 25 avril 12:10 Makefile.url2pkg~
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Overriding
drwxr-xr-x 2 rancune rancune 4096 1 mai 12:26 patches
-rw-r--r-- 1 rancune rancune 32 25 avril 12:41 PLIST
-rw-r--r-- 1 rancune rancune 0 1 mai 12:23 Tool
drwxr-xr-x 11 rancune rancune 4096 1 mai 12:23 workA new directory, “patches”, has been created. It contains our patch. However, this new file is missing from our file list in distinfo. To add it, we use the makepatchsum target, also abbreviated mkps:
$ bmake makepatchsum
$ cat distinfo
$NetBSD$
BLAKE2s (truefalse-v1.0.tar.gz) = 0ebf7a1aaef5f20a6c0df0a0cb38cc9ff7cf1d7d511e18c2ad5a4ec[...]
SHA512 (truefalse-v1.0.tar.gz) = 0be26cb93917249540e7d0d37805dcd4540382df91fafc9b9c48a15b[...]
Size (truefalse-v1.0.tar.gz) = 492 bytesOur new version of the package, including the patch, is ready! All that’s left is to test it:
$ bmake clean
===> Cleaning for truefalse-1.0
$ bmake update
===> Checking for vulnerabilities in truefalse-1.0
===> Deinstalling for truefalse-1.0
Running /home/rancune/pkg/sbin/pkg_delete -K /home/rancune/pkg/pkgdb -r truefalse-1.0
=> Bootstrap dependency digest>=20211023: found digest-20220214
=> Checksum BLAKE2s OK for truefalse-v1.0.tar.gz
=> Checksum SHA512 OK for truefalse-v1.0.tar.gz
===> Installing dependencies for truefalse-1.0
=> Tool dependency nbpatch-[0-9]*: found nbpatch-20151107
=> Tool dependency cwrappers>=20150314: found cwrappers-20220403
===> Checking for vulnerabilities in truefalse-1.0
===> Overriding tools for truefalse-1.0
===> Extracting for truefalse-1.0
===> Patching for truefalse-1.0
=> Applying pkgsrc patches for truefalse-1.0
=> Ignoring patchfile /home/rancune/pkgsrc/wip/truefalse/patches/patch-truefalse.c.orig
===> Creating toolchain wrappers for truefalse-1.0
===> Configuring for truefalse-1.0
===> Building for truefalse-1.0
cc -O2 -D_FORTIFY_SOURCE=2 -c truefalse.c
cc -Wl,-zrelro -Wl,-R/home/rancune/pkg/lib -o truefalse truefalse.o
===> Installing for truefalse-1.0
=> Generating pre-install file lists
=> Creating installation directories
=> Automatic manual page handling
=> Generating post-install file lists
=> Checking file-check results for truefalse-1.0
=> Creating binary package /home/rancune/pkgsrc/wip/truefalse/work/.packages/truefalse-1.0.tgz
===> Building binary package for truefalse-1.0
=> Creating binary package /home/rancune/pkgsrc/packages/All/truefalse-1.0.tgz
===> Installing binary package of truefalse-1.0
===> Cleaning for truefalse-1.0We can clearly see the patch being applied! And if we test, our program now works!
Proud as peacocks (and frankly, we have good reason to be, don’t we?), we can submit our patch upstream. Because – and this is an important part of today’s message – the packager plays an important role in the development of an application by actively participating in it. It is not unusual for a packager to “adopt” a project abandoned by its developers but still needed by their community!
It is with great joy that we see a [version 1.1][https://gitlab.com/iMil/truefalse/-/commits/v1.1] of truefalse appear, integrating our patch… \o/
truefalse v1.1 is out? Let’s go!
Since version 1.1 is out, we will update our package. I can already hear you: “What?? We have to redo everything?” Not at all! With pkgsrc, this will be painless!
We start by modifying our Makefile to switch to version 1.1. While we’re at it, let’s put the version number in a VERS variable – it will avoid mishaps!
# $NetBSD$
VERS= v1.1
DISTNAME= truefalse-${VERS}
PKGNAME= ${DISTNAME:S,-v,-,}
CATEGORIES= misc
MASTER_SITES= https://gitlab.com/iMil/truefalse/-/archive/${VERS}/
MAINTAINER= truefalse@rancune.org
HOMEPAGE= https://gitlab.com/iMil/truefalse/-/archive/${VERS}/
COMMENT= First attempt at making a package
LICENSE= original-bsd
INSTALLATION_DIRS= bin
do-install:
${INSTALL_PROGRAM} ${WRKSRC}/${PROGNAME} ${DESTDIR}/${PREFIX}/bin/${PROGNAME}
.include "../../mk/bsd.pkg.mk"From now on, the version 1.1 sources will be used:
$ bmake fetch
=> Bootstrap dependency digest>=20211023: found digest-20220214
=> Fetching truefalse-v1.1.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 487 100 487 0 0 1497 0 --:--:-- --:--:-- --:--:-- 1503We update the distinfo file so that the checksums for this new version are included:
$ bmake makesum
=> Bootstrap dependency digest>=20211023: found digest-20220214
$ cat distinfo
$NetBSD$
BLAKE2s (truefalse-v1.1.tar.gz) = 2f3f23cb1606ab1cec7e856db861d4d383c482273108307ae41516e[...]
SHA512 (truefalse-v1.1.tar.gz) = 4a8cad86ea98452e5c7f2a669d61c92447a9337aca8d444e2c261a6c[...]
Size (truefalse-v1.1.tar.gz) = 487 bytes
SHA1 (patch-truefalse.c) = 78e58a136994df49789451bfa5f667b183dfd80cWe can also remove our patch from the package, since it has been integrated upstream. We must then remember to also remove the patch checksums from the distinfo file:
$ rm -fr patches
$ bmake makepatchsumWe test… and it works!!!
$ bmake update clean
$ truefalseYou have new mail
A new day dawns: the birds are singing, the sun is shining, there’s a nice smell of coffee in the office… And you receive a new email: OOOOOOh, a version 2.0 of truefalse has been released!
You take a look at the sources, and disaster strikes:
#include <stdlib.h>
#include <glib.h>
int
main(int argc, char *argv[])
{
if (argc < 2) {
g_assert_true(argc < 2);
return EXIT_SUCCESS;
}
g_assert_false(argc < 2);
return EXIT_FAILURE;
}But… but… but???? Why??? Our developer decided to use glib?????
And this Makefile: why abandon bmake? We’re using the GNU version of make now????
CFLAGS= $(shell pkg-config --cflags glib-2.0)
LDFLAGS= $(shell pkg-config --libs glib-2.0)
truefalse: truefalse.c
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)We will have to deal with all of this… But that will be in the next video!
See you very soon,
Rancune.