Poudriere: A complete guide

This is my attempt at providing a guide on how to setup Poudriere to build your own binary packages for FreeBSD.

The release of FreeBSD 10 brought us pkg for binary package management together with the offical binary repositories. Until then, using any kind of 3rd-party software required building the package yourself. From FreeBSD 10 onwards, users can easily install binary packages using pkg install <package_name>. While this is nice to users that are new to FreeBSD (especially folks coming from the Linux world), anyone running at least a couple of production servers know that this comes with its own sets of problems. Most notably: The binary packages provided by the offical repositories might not have been compiled with options that are optimal for your needs and more importantly: Some features might not have been enabled. As an example: The net-mgmt/zabbix52-server package from the offical repositories has been compiled with support for MySQL instead of PostgreSQL. Furthermore, IPMI monitoring is disabled. In a situation like this there’s no way around compiling the port yourself. But there’s a catch: Mixing self-compiled packages from ports and the binary releases from the offical repositories is highly discouraged. There is no mechanism that handles dependencies between these two systems. While this might not be a big deal for “end-user” software this can certainly lead to very nasty situations that are hard to identify and fix.

Therefore, even after the introduction of the offical binary repositories there are plenty of situations where building your own ports is beneficial. However, building the ports on every single host can be quite a hassle in itself. The solution: Build the ports you need on one machine and have all other machines consuming these binary packages. Poudriere can be used to built the packages. The built packages can still be managed through pkg on the clients/consumers.

Precursor

While it’s possible to run poudriere on any machine running FreeBSD (such as a desktop/laptop) it’s usually implicitly necessary to run poudriere on a proper server. Builds will take a long time - especially when building any kind of “desktop software packages” - and can easily span multiple days. Furthermore, performing builds in parallel will require a decent amount of memory.

I decided setup my poudriere server in a bhyve VM due to the following reasoning:

  • This allows me to easily migrate the entire poudriere server with all configurations and built packages to a different host should that ever be desired or necessary.
  • Easier resource management
  • Poudriere uses jails for building packages. Jails can only be the same or older versions than the host they are running on. I want my physical server to run -RELEASE which wouldn’t allow me to build packages for -STABLE.

However, this is completely transparent to this guide. You can follow this guide no matter whether you’re running poudriere bare-metal or in any kind of VM (doesn’t need to be a bhyve VM!).

It is beneficial to use poudriere on ZFS as poudriere will create, manage & destroy jails which is a lot less I/O intensive on ZFS than UFS.

Setting up the server

Setting up the poudriere server consists of the following steps:

  1. Generate private key and public SSL certificate for signing the packages
  2. Install & configure poudriere
  3. Install & configure a web server to deliver the built packages & web-based reporting

In my particular scenario, I will use freebsd.binaries.insane.engineer as the URL for serving built packages (and the web interface). When following this guide, replace that URL with whatever URL you’re going to use.

Generating SSL certificate

We will configure poudriere sign the built packages. A client/consumer will check this signature to verify the integrity/authenticity of the binaries before installing them.

We need to generate a key and SSL certificate. These are usually located in /usr/local/etc/ssl/. Lets make sure the directories exist:

mkdir -p /usr/local/etc/ssl/{certs,keys}

Next, we generate a 4096-bit RSA key:

openssl genrsa -out /usr/local/etc/ssl/keys/poudriere.key 4096

The private key must be kept secret as anyone in posession of the private key can potentially provide us with malicious packages:

chmod 0600 /usr/local/etc/ssl/keys
chmod 0600 /usr/local/etc/ssl/keys/poudriere.key

The public certificate gets generated from the private key:

openssl rsa -in /usr/local/etc/ssl/keys/poudriere.key -pubout -out /usr/local/etc/ssl/certs/poudriere.cert

The private key & public certificate are now ready.

Setting up poudriere

Poudriere needs to be installed on the server. The easiest way of doing this is to simply pull the pre-built package from the official FreeBSD pkg repository:

pkg install poudriere

Poudriere’s main configuration file can be found at /usr/local/etc/poudriere.conf. In there, we want to set the following configuration options for a basic setup:

NO_ZFS=no
ZPOOL=zroot
ZROOTFS=/poudriere
FREEBSD_HOST=https://download.freebsd.org
BASEFS=/usr/local/poudriere
POUDRIERE_DATA=$(BASEFS)/data
USE_TEMPFS=yes
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://freebsd.binaries.insane.engineer

The configuration file is fairly-well inline documented so there’s little benefit in repeating this information here. The other options seem to have decent defaults for a basic setup.

Next it’s time to setup the build environment(s). In this guide, we will only setup a build environment for FreeBSD 13.0 x64 binaries. Adding different build environments is easy by tweaking the various parameters of the following commands. We start by creating a jail for our 13.0x64 binaries. Poudriere manages its own jails which makes this process rather easy:

poudriere jail -c -j 130Ramd64 -v 13.0-RELEASE -a amd64

This creates a jail named 130Ramd64 with FreeBSD 13.0-RELEASE as the base targetting amd64 machines. When naming your jail, look at the CAVEATS section of man poudriere-jail (currently the most notable restriction is that the name must not contain a period '.'.

The next step is to install the ports structure. Ports management takes place through the poudriere ports command. In this example we will build the latest ports available. Therefore, we’re going to name our ports structure HEAD (this would correspond to the ’latest’ binary train of the official binary repositories):

poudriere ports -c -m git+https -B main -p HEAD

The installation & configuration of poudriere is now complete.

Packages can now be build using poudriere-bulk:

poudriere bulk -j 130Ramd64 -p HEAD -f path/to/your/port_list

We will automate this process further below.

The -f option of poudriere-bulk accepts a path to a file containing a list of all ports that should be build. The format of the file is straight forward: Each line lists a port using the category/port notation.

Setting up the webserver

Clients need a way of accessing the packages that poudriere built. One way of doing this is setting up a webserver. In this case, I am going to use NGINX for this purpose. Furthermore, poudriere comes with a (read-only) web frontend to collect information about the ongoing and finished building processes.

NGINX can be installed through pkg as well:

pkg install nginx

Copy the nginx sample configuration file that ships with poudrier and modify it to fit your needs (usually at least the listen and server_name parameters).

Packages can now be accessed via http://freebsd.binaries.insane.engineer/packages. Furthermore, the handy web-frontend is accessible under http://freebsd.binaries.insane.engineer.

Depending your situation, you might want to setup NGINX to use TLS. In my scenario, the NGINX server in question is located behind a net/haproxy TLS terminator so this is not necessary. If you’re interested into this sort of thing, you might want to checkout my blog post on setting up a TLS terminator on FreeBSD.

Configuring clients

At this point we have a working server which uses poudriere to build ports and makes them accessible via NGINX. We are now ready to setup one or more clients to start using our own binary repository through pkg. The commands shown in this section are to be executed on a client that should consume our custom built packages - NOT the poudriere server.

Our packages get signed via an RSA key. We want to have to have the poudriere’s poublic key available on the client so it can verify the integrity/authenticity of the packages server by our poudriere server.

Place poudriere’s public key under /usr/local/ssl/certs/, for example: /usr/local/etc/ssl/certs/poudriere.cert. I’ll leave it up to the reader on how to get the public key from the poudriere server to the client. You may grab it via SSH or copy it through other means.

Next, the pkg repositories need to be modified. By default, there is only one repository registered: The official pkg.freebsd.org. Opening /etc/pkg/FreeBSD.conf reveals the following information:

# $FreeBSD$
# To disable this repository, instead of modifying or removing this file, create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf

FreeBSD: {
	url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
	mirror_type: "srv",
	signature_type: "fingerprints",
	finterprints: "/usr/share/keys/pkg",
	enabled: yes
}

Well, that is pretty handy information. Let’s follow those instructions:

mkdir -p /usr/local/etc/pkg/repos
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf

If you now execute pkg update -f you should be informed that there are no active repositories configured:

jbo@x1carbon ~> pkg update -f
No active remote repositories configured.

This confirms that we successfully disabled the official pkg repository. We can now proceed to add our own repository by creating /usr/local/etc/pkg/repos/insaneengineer.conf:

InsaneEngineer: {
	enabled: yes,
	url: "https://freebsd.binaries.insane.engineer/packages/130Ramd64-HEAD",
	mirror_type: "https",
	signature_type: "pubkey",
	pubkey: "/usr/local/etc/ssl/certs/poudriere.cert"
}

Again: You want to name this in a way that matches your setup. I call my repository InsaneEngineer as that is where my poudriere server is accessible through. Do not forget to adjust the url!

Now we can try to grab the packages list by executing pkg update -f:

Updating InsaneEngineer repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.txz: .......... done
Processing entries: .......... done
InsaneEngineer repository update completed. 986 packages processed.
All repositories are up to date.

Success!

At the moment, the system is still using the packages from the official pkg mirror. Now that we updated our packages list which is now exclusively pointing to our own repository, we can “migrate” to our repository:

pkg upgrade

And that’s it. The client system is now consuming the packages from our poudriere server. From this point on nothing different needs to happen. Whenever you want to update your packages on the client, just run pkg update followed by pkg upgrade as you did with the public packages before.

Automating builds

At this point we have a working poudriere server which builds packages and serves them via HTTPS as well as clients that consume these packages. The next step is to automate the build of packages.

In my case, I want a shell script which accepts the jail as a parameter, updates the ports tree and then builds all repositories for that jail. Here’s what I came up with:

#!/bin/sh

###
### Settings
###
PATH_PORTLIST="/usr/local/etc/poudriere.d/port_list.txt"
REPOS="HEAD"
BIN_POUDRIERE="/usr/local/bin/poudriere"

###
### Parameters
###
JAIL=$1

#####################################################################
### Don't change stuff below here unless you know what you're doing #
#####################################################################

#
# Don't run this script more than once at a time.
#
SCRIPTNAME=`basename "$0"`

# Check for running script
STATUS=`ps -ax | grep "$SCRIPTNAME" | grep -v grep | wc -l`

# Compare the two because ` creates a sub-process
if [ "$STATUS" -gt 2 ]; then
	echo "already running."
	exit 0
fi

#
# Build function
#
poudriere_build() {
	for REPO in $REPOS; do
		$BIN_POUDRIERE bulk -j "$JAIL" -p "$REPO" -f $PATH_PORTLIST
	done
}

#
# Update repos function
#
repos_update() {
	echo "repos: $REPOS"
	for REPO in $REPOS; do
		echo "repo: $REPO"
		echo "[$SCRIPTNAME] Updating ports tree... $REPO"

		$BIN_POUDRIERE ports -p "$REPO" -u

		if [ $? -ne 0 ]; then
			echo "    Error updating ports tree."
			exit 1
		fi
	done
}

#
# Script main stuff
#
echo "updating repositories..."
repos_update

echo "starting build..."
poudriere_build

echo "cleaning distfiles..."
#$BIN_POUDRIERE distclean -p "$REPOS" -f "$PORTLIST" -y

exit 0

In my case, I setup some cron jobs which call this script with the appropriate argument for each jail.

comments powered by Disqus