Cross compiling the FreeBSD Kernel

Wow, two post in one day already ;-)

There are two things I'd like to note. First, I noticed that cross-compiling seems a major issue for me. Don't know why that is.

Second, I need to remember Warner Losh's post on cross compiling FreeBSD. Essentialy, the procedure is:

$ export TARGET=powerpc
$ export TARGET_ARCH=powerpc
$ make kernel-toolchain
$ make buildkernel

Addendum: Unfortunately, this procedure works only on architectures already supported by FreeBSD and its build system. Therefore, it doesn't work for me. So here's the full story on how I got FreeBSD to at least compile.

Building the Toolchain

Building the toolchain is pretty straight forward. I've already written about how to build a cross compiler. On FreeBSD however, four things are different.

• The target is powerpc64-unknown-freebsd. I don't know if powerpc64-unknown-elf would have worked, too.
• The target is new, so a patch to the binutils sources is required.
• The GNU MP Bignum Library (GMP) is required. I used version GMP 4.2.1 and installed it in $PREFIX
• Finally, the MPFT Library needs to be built. I used version MPFT 2.3.0 and installed it in $PREFIX

Note that those steps have to be performed before the compiler is built. Since I didn't install the libraries in the standard locations, the LD_LIBRARY_PATH variable needs to be set before the compiler can be used:

$ export LD_LIBRARY_PATH=$PREFIX/lib

Building the Kernel

The basic procedure of building a kernel is outlined in the FreeBSD Developer's Handbook. Provided that the cross compiler has been installed in $PREFIX, these steps are required:

$ export MACHINE=powerpc64
$ export MACHINE_ARCH=powerpc64
$ export CC=${PREFIX}/${TARGET_PREFIX}-gcc
$ export NM=${PREFIX}/${TARGET_PREFIX}-nm
$ export LD=${PREFIX}/${TARGET_PREFIX}-ld
$ export SYSTEM_LD=${LD}
$ export OBJCOPY=${PREFIX}/${TARGET_PREFIX}-objcopy
$ cd /usr/src/sys/powerpc64/conf
$ config KERNCONF
$ cd ../compile/KERNCONF
$ make cleandepend && make depend
$ make kernel

Oh, of course this doesn't work with the stock FreeBSD sources. Instead, my FreeBSD 64-Bit PowerPC Mercurial repository needs to be used. Note that for convenience reasons, that tree includes a crossenv script which, when sourced, sets up the required environment variables.

Linux KVM (kvm-70) on IBM Open Client 2.2

The Linux kernel-based virtual machine (KVM) is a great way for virtualization on computers running Linux. It requires virtualization support by the host processor (most modern x86 CPUs have this) and a kernel module. The kernel module can be built from the KVM sources.

Unfortunately, compiling the module on the IBM Open Client 2.2 distribution doesn't work out of the box. Instead, a patch is required. The patch is an extended version of this commit to the KVM repository and applies against the KVM-70 release tar ball.

Networking

The KVM networking documentation lists brctl(8) and tunctl(8) as requirements for a bridge between the host and the guest. On the Open Client distribution, the brctl utility is part of the bridge-utils package whereas the tunctl tool is part of uml-utils - on other distributions, that is. However, there is a Fedora Core 9 package available which seems to work.

Before starting the KVM guest, make sure that the tun kernel module is loaded. These are the steps I use to start the guest:

$ sudo modprobe tun
$ MACADDR=`genmac`
$ sudo qemu-system-x86_64 -hda freebsd-7.0.img \
   -net nic,macaddr=$MACADDR -net tap,script=qemu-ifup

Note that the genmac and qemu-ifup scripts are the examples from the KVM documentation.

NAT on the bridge

Because I cannot put the KVM guest on the host network, I need to do NAT on the host. I've found this quick tutorial on NAT with iptables. The four steps are:

# echo 1 > /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# iptables -A FORWARD -i eth0 -o tap0 -m state \
     --state RELATED,ESTABLISHED -j ACCEPT
# iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT

Also, make sure the tap0 interface has an IP address:

$ sudo ifconfig tap0 192.168.0.1/24

PowerPC Device Tree Compiler

The Device Tree Compiler (DTC) project is hosted at OzLabs. The website seems to be unavailable at the moment, but the git repository at git://ozlabs.org/srv/projects/dtc/dtc.git seems to work.

Cross-building the tools works fine. This is what I did:

$ export CC=ppu-gcc
$ make

This will create the dtc and ftdump tools which can then be copied to the target machine.

Cross-compiling the Linux kernel

I need to cross-compile a PowerPC Linux kernel on an x86 laptop. I've found instructions on how to compile (not cross-compile) the Linux kernel at this website. Further, there is a post to a mailing list here which shows how to cros-compile the kernel. The mailing list post mentions a ccontrol file, but I have no clue what that is. Luckily I've found this blog post, which seems to be more accurate.

Building a PowerPC Cross Compiler

I need to build my own cross compiler which will run on i386 and produce 64-Bit PowerPC Binaries. I've found a pretty neat introduction to building a cross compiler on IBM's developerWorks site (registration required). The tutorial isn't a step-by-step guide, but it helped me a lot.

The basic procedure for building a cross-compiler is:

• Obtain headers specific to the operating system
• Build binutils for the target platform
• Build a bootstrap compiler
• Build a standard C library for the target
• Build the final cross-compiler, using the C library just built

The developerWorks tutorial doesn't mention this, but the first three steps can easily be run in parallel. Anyways, before starting, I've set these environment variables:

$ export TARGET=powerpc64-unknown-linux-gnu
$ export PREFIX=/opt/crosscompiler
$ export TARGET_PREFIX=$PREFIX/$TARGET
$ export PATH=$PATH:$PREFIX/bin

Obtaining Linux-specific Headers

I've followed the developerWorks tutorial on this one: Downloaded and extracting the Linux kernel sources, then copied the relevant files. Here are the commands I ran:

$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.9.tar.bz2
$ tar xvjf linux-2.6.25.9.tar.bz2
$ cd linux-2.6.25.9
$ make ARCH=powerpc CROSS_COMPILE=powerpc64-linux- menuconfig
(configure options, but tweaking isn't neccessary)
$ mkdir -p $TARGET_PREFIX/include
$ cp -r include/linux $TARGET_PREFIX/include
$ cp -r include/asm-powerpc $TARGET_PRFIX/include/asm
$ cp -r include/asm-generic $TARGET_PREFIX/include

If you read to the end of this post, then you'll realize that this step wouldn't be required (for now).

Building GNU binutils

I'm using GNU binutils 2.18, available from the GNU website. These are the steps required to build binutils.

$ wget 
$ tar xjvf
$ ./configure --prefix=$PREFIX --target=$TARGET --disable-nls -v
$ make all
$ make install

While building binutils did take a while, it wasn't as long as the tutorial makes you believe. On a IBM Thinkpad T60p built around a Centrino Duo CPU running at 2.16 MHz it took less than 10 minutes. Also note the last command ("make install"), which is missing from the developerWorks tutorial.

Building a bootstrap compiler

For my project I need GCC 4.x, the latest version at the time of writing is 4.3.1 which is available from a GNU mirror near you. Downloading and extracting is easy:

$ wget ftp://ftp.gwdg.de/pub/misc/gcc/releases/gcc-4.3.1/gcc-4.3.1.tar.bz2
$ tar xjvf gcc-4.3.1.tar.bz2
$ cd gcc-4.3.1

Here are the steps required to build a bootstrap compiler.

$ ./configure --prefix=$PREFIX --target=$TARGET --without-headers \
  --with-newlib -v
$ make all-gcc
$ make install-gcc

This took longer than building binutils, however it took less than 30 minutes (as opposed to the hours the tutorial talks about).

Building the GNU C Library (glibc)

$ CC=${TARGET}-gcc ./configure --target=$TARGET --prefix=$PREFIX \
  --with-headers=${TARGET_PREFIX}/include

Unfortunately, this command failed with the following error:

(...)
checking whether __attribute__((visibility())) is supported... no
configure: error: compiler support for visibility attribute is required

However, this isn't important as I won't need a standard C library for now - I'm building with -ffreestanding and -nostdlib anyways. Therefore I've decided that I won't pursue this futher but may come back later.