------------------------------------------------------------------------------ 1. Introduction ------------------------------------------------------------------------------ This document explains a way to build a split-objs enabled version of ghc-api. The description contains code blocks are delimited by >>> and <<< as follows: >>> code here <<< Unless indicated otherwise, they contain shell commands. Required tools: - a recent build of ghc's head, installed - a recent Cabal build (I used the darcs version; I had trouble with 1.3.2.) I tested the procedure on a 32 bits x86 Linux system. ------------------------------------------------------------------------------ 2. Building ------------------------------------------------------------------------------ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 2.1 Preparation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - pick a working directory, $WORK - put current ghc sources in $WORK/ghc (it doesn't really matter where they are, but the instructions below assume this location) - unpack ghc-api-test.tar.bz2 in $WORK, this creates a subdirectory $WORK/ghc-api-test - prepare a few things in the ghc build tree. we need: - libffi - some preprocessing in the compiler subdirectory. basically, we simulate a stage2 build. - the previous step requires the mkdirhier and genprimopcode utils >>> cd $WORK/ghc ./boot ./configure ( cd utils/mkdirhier; make ) ( cd utils/genprimopcode; make depend; make ) ( cd libffi; make ) cd compiler make stage=2 stage2/ghc_boot_platform.h make main/Config.hs make stage=2 \ primop-data-decl.hs-incl \ primop-tag.hs-incl \ primop-list.hs-incl \ primop-has-side-effects.hs-incl \ primop-out-of-line.hs-incl \ primop-commutable.hs-incl \ primop-needs-wrapper.hs-incl \ primop-can-fail.hs-incl \ primop-strictness.hs-incl \ primop-primop-info.hs-incl make parser/Parser.y cd .. <<< - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 2.2 Creating the source tree. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Next, copy all the prepared files to our build directory. >>> cd $WORK/ghc-api-test cp -r ../ghc/libffi . cp -r ../ghc/compiler/* . cp -r ../ghc/includes . patch -p1 < ghc-api-test.patch <<< Now edit main/Config.hs and replace the current date (for example, 20080514) with the date of your ghc build (see ghc --version, for example 20080507) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 2.3 Building - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - We can now build this (almost) like any cabal package. There's a single hs-boot file that Cabal doesn't pick up. >>> cd $WORK/ghc-api-test mkdir -p dist/build cp parser/HaddockLex.hs-boot dist/build/ runghc Setup.lhs config -O2 --enable-split-objs runghc Setup.lhs build runghc Setup.lhs install ghc-pkg hide ghc-api-test <<< ------------------------------------------------------------------------------ 3. Experiments ------------------------------------------------------------------------------ All the experiments are related to plugin loading. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 3.1 A simple ghc-api based plugin loader. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >>> cd $WORK/ghc-api-test/experiments/plugin-ghc-api ghc --make Plugin.hs ghc --make -no-recomp Main.hs -package ghc strip Main mv Main Main.ghc ghc --make -no-recomp Main.hs -package ghc-api-test strip Main mv Main Main.ghc-api-test <<< Result: Main.ghc-api-test is only marginally smaller than Main.ghc. This is hardly surprising, because the code actually compiles an expression, which pulls most of the compiler's logic in, parsing, type checking, and code generation. Note: With stripped objects, the second build may run out of memory. To fix this, you can try passing the --hash-size=50 option to ld. (FIXME: how?) Alternatively, you can apply the patch binutils-2.18-tune-bfd-hash.patch from the tar-ball to binutils and rebuild ld, which brings the memory usage down to less than 400 MB (which is still crazy, but linking about 40000 .o files has its price) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 3.2 A ghc-api based .hi file loader. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >>> cd $WORK/ghc-api-test/experiments/hi-loader ghc --make Plugin.hs ghc --make -no-recomp Main.hs -package ghc strip Main mv Main Main.ghc ghc --make -no-recomp Main.hs -package ghc-api-test strip Main mv Main Main.ghc-api-test <<< Result: Main.ghc-api-test is about 90% smaller than Main.ghc. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 3.3 hs-plugins - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This requires two versions of hs-plugins, compiled with ghc and ghc-api-test, respectivly. Building then works similar to the previous examples. Again, the ghc-api-test version is about 85% lot smaller than the ghc one. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 3.4 Numbers - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - section | name | executable size | | ghc | ghc-api-test 3.1 | plugin-ghc-api | 13273384 | 11457212 3.2 | hi-loader | 12941516 | 1213288 3.3 | plugin-plugins | 13479292 | 1813392 ------------------------------------------------------------------------------ 4. Summary ------------------------------------------------------------------------------ Enabling --split-objs for ghc-api is tricky, due to the massive number of objects being created; however, it may be worth the effort for lightweight plugin loading libraries. It's obviously not worthwhile for the compiler ghc itself. Bertram Felgenhauer History: 2008-05-17 Initial version