Steve Price

steve@FreeBSD.org

$FreeBSD: doc/en_US.ISO8859-1/articles/releng-packages/article.sgml,v 1.1 2002/02/25 14:33:46 murray Exp $ This paper describes the approach used by the FreeBSD release engineering team to produce a high quality package set suitable for official FreeBSD release media. This document is a work in progress, but eventually it will cover the process used to build a clean package set on the FreeBSD.org "Ports Cluster", how to configure any other set of machines as a ports cluster, how to split up the packages for the release media, and how to verify that a package set is consistent. The FreeBSD Ports collection is a collection of over &os.numports; third-party software packages available for FreeBSD. The ports team (portmgr@FreeBSD.org) is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany a given FreeBSD release. In order to provide a consistent set of third-party packages for FreeBSD releases, every port is built in a separate chroot environment, starting with an empty /usr/local and /usr/X11R6. The requisite dependencies are installed as packages before the build proceeds. This enforces consistency in the package build process. By starting the package build in a pristine environment, we can assure that the package metadata (such as required dependencies) is accurate, and so we will never generate packages that might work on some systems and not on others depending on what software was previously installed. The Ports Cluster for the x86 architecture currently consists of a master node (Dual Pentium III 733Mhz) and 8 slave nodes (Pentium III 800Mhz) to do the actual package builds. With this configuration, a complete package build takes over 24 hours. These machines are co-located with the other FreeBSD Project equipment at Yahoo's corner of Exodus in Santa Clara, CA. The Ports Cluster for the Alpha architecture consists of 7 PWS 500A machines donated by Compaq and also co-located with Yahoo's facilities. For FreeBSD 4.4 over 4.1 gigabytes of packages were created. This causes a problem for CDROM distributions because we would like to ship as many packages as possible without making the user insert another disc to satisfy dependencies. The solution is to create clusters of like packages with similar dependencies and group these onto specific discs. This section describes the software and methodology used to create those package sets for the official FreeBSD release discs. First off you'll need to get a copy of the tarball from the following URL : Copy thisc archive to a machine that has enough free HD space to hold 2 to 3 times the size of the package set that you wish to split. The tarball will extract into the current working directory so make sure you've created one suitably named directory for the release you are working on. After you've extracted the files, you will notice the following files : config This file contains the free space on each disc and whether packages, distfiles, or both are allowed on any given disc. The first column is the disc name. It must be of the form disc[0-9a-z]. Currently it is setup to allow for 10 discs (4 for the release set and 6 for the toolkit). There's an implied extra disc called scratch where all of the remaining distfiles/packages land if they don't fit elsewhere. The second column can be either a 1 or 0 where 1 says that it is okay to place packages on this disc. The third column works the same way except that it controls whether distfiles are placed on this disc. The last column denotes the number of bytes of free space on a disc. doit.sh This is the workhorse. Once you have all the files in place and things properly configured this script directs the process of splitting packages. Beware it is interactive so you need to keep an eye on it as it runs. More details on what happens in this script will follow. scripts/checkdeps.pl Makes sure all packages dependencies are satisfied given an INDEX file and a directory of packages. scripts/oneshot.pl This is where all the magic (and I use that term loosely as it is mostly just a brute force approach) happens. Given a list of required packages for each disc and a set of packages/distfiles this is the script that places a package or distfile on a disc along with all of its dependencies. scripts/print-cdrom-packages.sh This file is a copy of src/release/scripts/print-cdrom-packages.sh from the release you are working on. scripts/scrubindex.pl This script removes lines from an INDEX file for packages that aren't present. It also removes the XFree86 dependencies. NOTE: you'll need to tweak the value of the xdep variable to make sure the version number is correct. scripts/setup.sh This is a helper script that I use on the bento cluster to grab a copy of the ports tree and the matching set of the packages/distfiles. Here's a checklist of things you'll need to check or configure before going any further. Edit config to denote the number of discs you have, their sizes, and whether you want them want to contain packages, distfiles, both, or neither. Make sure you remove the gen directory if there's an old one laying around. This directory contains working files that will only be valid for the current split. On your first pass through a split it is best to fake the copying of packages and distfiles. This will save both time and diskspace while you do a couple of trial runs to make sure things fit, etc. In the scripts/oneshot.pl set the fake variable to 1 and instead of actually copying the files it will &man.touch.1; them. Be sure you turn this off or set fake to 0 before you give the resultant discs to the person that will be mastering the discs otherwise they'll get a directory full of zero-sized files. Make sure you have a recent copy of the print-cdrom-packages.sh and that it is from the correct release. Check to make sure the XFree86 dependency in scripts/scrubindex.pl has the correct version number. You'll also need to make sure this value is correct in doit.sh as well. Next you'll need to get a copy of the ports tree, packages, and distfiles from a recent build on the package cluster. See the scripts/setup.sh for a working example but essentially here's what needs to be done. Grab a copy of ports.tar.gz and extract it into the ports directory alongside doit.sh and the scripts directory. Remove the packages/distfiles directories or symlinks. Bento has these as symlinks and you'll have mixed results if you don't get rid of them before proceeding. Create a new ports/packages directory and copy the package set from the package building cluster. Create a new ports/distfiles directory and copy the distfiles from the package building cluster. NOTE: if you don't want any distfiles simply create the directory and leave it empty. This directory must be present even if it doesn't contain anything. Now we're finally ready for the fun task of actually splitting the packages. You start the processing by running ./doit.sh. The first time you run it here's what it does. Create a list of the restricted (can't be on the master FTP site) ports. Asks you if you'd like to remove the restricted ports. Most of the time you'll want to answer (y)es here. Create a list of the packages/distfiles that can't be put on the discs. Asks you if you'd like to remove the non-cdromable packages/distfiles. Most of the time you'll want to answer (y)es here. Copies the INDEX from the ports directory to the gen directory. In doing so it removes the lines for ports where the packages don't exist. It also checks to make sure that all of the required dependency packages are present. Create a list of packages that are required on each disc. Asks you if you'd like to populate the discs. After populating each disc it will check for missing dependencies, scrub the INDEX file, and create the CHECKSUM.MD5 file. Check to make sure the required packages made it on each disc and gives you a summary of the sizes of each disc. After going through this the first time if you are lucky enough that all of the required packages built and fit on each disc. All you need to do is set fake to 0 in scripts/oneshot.pl and re-run ./doit.sh. The second and subsequent times around it will skip steps 1-5 above. If you want to re-run any of those steps refer to doit.sh for which files need to be removed to not short-circuit those steps. If you want to repeat all of these steps then the easiest way is to rm -rf gen. Upon successful completion the packages/distfiles will be in the disc* directories and the leftover will be in the scratch directory. What to do if things go wrong? Here's some common gotchas and workarounds. Missing required packages This is a pretty common occurrence. You'll either need to wait for a new set of packages where the missing packages were built or get someone to re-start the package build for you. DO NOT attempt to build the missing packages on your own machine and add them into the fray. While you might be able to get away with this if you are extremely careful the vast majority of the time you'll miss some little detail and the simple process of adding a package could make hundreds of others come up mysteriously broken. Required packages won't fit This happens on occasion too and is relatively easy to fix. Simply edit print-cdrom-packages.sh to move packages around until they fit. Yes this is an iterative process and one of the reasons why you should enable fake in scripts/oneshot.pl until you've gotten things the way you want them. Re-run ./doit.sh after you made your adjustments. Required packages not on the right (or any) disc This usually means you didn't add them to print-cdrom-packages.sh or you put them on the wrong disc. This script is the gospel by which this whole process determines where a package must be. If you want to force a package to land on a particular disc this is the only way to ensure that it will happen. If you get completely stuck and can't figure out why things are borked or how to fix them then email &a.steve; for assistance.