HOWTO - Blue Gene LAPACK/CBLAS library build
Building LAPACK and CBLAS on Blue GeneNils Smeds, IBM Sweden. <nils.smeds> at <se.ibm.com>
Mattias Claesson, PDC, KTH. <slas> at <pdc.kth.se>
The information in this document is provided as is. No warranties or support is given for the procedure described here. You are free to use this information and the patches here with no obligations (but we of course value to be referenced when suitable). The LAPACK, CCI and CBLAS code base is available from NetLib. This document does not change or modify your rights and obligations in respect to those packages. To summarize, this has worked for us (to the extent that we have tested it). It may also work for you. If it does we would be happy to hear about it. If it does not we are interested in hearing that too.
- NetLib LAPACK repository
- NetLib CCI repository
- Our collection of files for the build BlueGene-LAPACK-buildpack.tar.gz
tar zxvf lapack-lite-3.1.1.tgz mv lapack-lite-3.1.1 LAPACKUnpack the cci package into the LAPACK directory. Note that CCI will put files in subdirectory LAPACK
tar zxvf cci.tgzEnter the LAPACK tree and apply the CCI modifications
cd LAPACK ./cciThis prints some output that ends with something like the following:
----8<-------8<-------8<-------8<-------8<-------8<--- If this script fails after this point, edit the make.inc file and make sure that BLASLIB defines the correct ESSL library. The default is -lessl. If the make.inc is changed, it will be necessary to remove the object files in LAPACK/SRC and type "make lapacklib" in the LAPACK directory. Makefile:7: make.inc: No such file or directory make: *** No rule to make target `make.inc'. Stop. CCI installation complete. Enjoy the enhanced performance of your IBM RS6K!! ----8<-------8<-------8<-------8<-------8<-------8<---Download the BlueGene build pack and install it in the LAPACK directory. Unpack it as follows:
tar -zxvf BlueGene-LAPACK-buildpack.tar.gzNow, insert modified files (source code) into the build tree
tar xvf fixed-files.tarUse the provided file make.inc.BGP or make.inc.BGL and put it in LAPACK/INSTALL. Choose the version that matches your system
cp INSTALL/make.inc.BGP make.incEdit the BLASLIB entry in the file to match your installation of ESSL.
Build the whole package. Note that due to the build trying to execute code on the compile host we need to build in three separate steps.
cd SRC make -k TIMER=RTC 2>&1 | tee ../MAKE-lapacklib.log cd .. make -k TIMER=RTC lib lapack_testing 2>&1 | tee MAKE-part2.log cd TESTING make -k TIMER=RTC 2>&1 | tee ../MAKE-testing.log cd ..Note the -k option to prevent the build to abort when it tries to run the generated Blue Gene binaries on the build system.
Testing the build.The LAPACK test procedure requires end-of-file to be handled on stdin. This appears to not always work. For this reason we have modified the test routines to accept a magic input line (sentinel) to denote end-of-file. Modify the input files to have the sentinel in them
./fix-input-filesSubmit the testing job (do not forget to edit the batch script to fit your site first). For BG/L runs there is a similar script llrun-bgl.sh.
cd TESTING llsubmit llrun.shInspect the output
../SummarizeRuns.bash Errors were 371 out of 3161696.None of the errors appears to us be alarming. They all relate to the level of accuracy in the computed answer. Raising the acceptance threshold in the input files makes the errors go away. In our experiments -O3 -qstrict gave similar accuracy to -O2. For this reason we chose to use -O3 -qstrict in the final file.
:::::::::::::: AllErrorsSummary-O2 :::::::::::::: Errors were 369 out of 3161696. :::::::::::::: AllErrorsSummary-O3-qnostrict :::::::::::::: Errors were 10857 out of 2126300. :::::::::::::: AllErrorsSummary-O3-qstrict :::::::::::::: Errors were 371 out of 3161696. /gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=default Errors were 371 out of 3161696. /gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=100.0 Errors were 278 out of 3161696. /gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=500.0
- Our collection of files for the build BlueGene-CBLAS-buildpack.tar.gz
tar zxvf cblas.tgz cd CBLASDownload our BlueGene CBLAS build pack into the CBLAS directory. Unpack the file
tar -zxvf BlueGene-CBLAS-buildpack.tar.gzThe tar file will create a soft link to Makefile.BGP. If you are on a BG/L system redirect this link to point to Makefile.BGL instead.
Edit Makefile.in. CBDIR and BLLIB are typically needed to be modified. A good value for CBDIR is the root of the CBLAS source tree. BLLIB needs to be adjusted to match your installation of ESSL.
Patch the test source code to work around a possible stdin EOF problem on Blue Gene and also to turn off error code check tests in the test examples. This is because by default ESSL exits with an error message on erroneous input.
patch -p0 < cblas-testing-bgp.patchBuild the libcblas_bgp.a library
make allAdd the file sentinel that the patch file enables to input files
./fix-input-filesRun the tests
cd testingEdit the submit script to match your site before submission. There is an alternative script for BG/L named llruntest-bgl.sh.