HOWTO - Blue Gene LAPACK/CBLAS library build

Building LAPACK and CBLAS on Blue Gene

Nils Smeds, IBM Sweden. <nils.smeds> at <se.ibm.com>
Mattias Claesson, PDC, KTH. <slas> at <pdc.kth.se>

The information in this document is provided as is. No warranties or support is given for the procedure described here. You are free to use this information and the patches here with no obligations (but we of course value to be referenced when suitable). The LAPACK, CCI and CBLAS code base is available from NetLib. This document does not change or modify your rights and obligations in respect to those packages. To summarize, this has worked for us (to the extent that we have tested it). It may also work for you. If it does we would be happy to hear about it. If it does not we are interested in hearing that too.

• Building LAPACK and CBLAS on Blue Gene
• LAPACK
• CBLAS

LAPACK

NetLib LAPACK repository

http://www.netlib.org/lapack
http://www.netlib.org/lapack/lapack-lite-3.1.1.tgz

NetLib CCI repository

http://www.netlib.org/lapack/essl
http://www.netlib.org/lapack/essl/cci.tgz

Our collection of files for the build BlueGene-LAPACK-buildpack.tar.gz

Unpack the LAPACK sources and rename them to LAPACK

tar zxvf lapack-lite-3.1.1.tgz
mv lapack-lite-3.1.1 LAPACK

Unpack the cci package into the LAPACK directory. Note that CCI will put files in subdirectory LAPACK

tar zxvf cci.tgz

Enter the LAPACK tree and apply the CCI modifications

cd LAPACK
./cci

This prints some output that ends with something like the following:

----8<-------8<-------8<-------8<-------8<-------8<---
If this script fails after this point,
edit the make.inc file and make sure that
BLASLIB defines the correct ESSL library.
The default is -lessl.
If the make.inc is changed, it will be
necessary to remove the object files in
LAPACK/SRC and type "make lapacklib" in
the LAPACK directory.

Makefile:7: make.inc: No such file or directory
make: *** No rule to make target `make.inc'.  Stop.
CCI installation complete.
Enjoy the enhanced performance of your IBM RS6K!!
----8<-------8<-------8<-------8<-------8<-------8<---

Download the BlueGene build pack and install it in the LAPACK directory. Unpack it as follows:

tar -zxvf BlueGene-LAPACK-buildpack.tar.gz

Now, insert modified files (source code) into the build tree

tar xvf fixed-files.tar

Use the provided file make.inc.BGP or make.inc.BGL and put it in LAPACK/INSTALL. Choose the version that matches your system

cp INSTALL/make.inc.BGP make.inc

Edit the BLASLIB entry in the file to match your installation of ESSL.

Build the whole package. Note that due to the build trying to execute code on the compile host we need to build in three separate steps.

cd SRC
make -k TIMER=RTC 2>&1 | tee ../MAKE-lapacklib.log
cd ..
make -k TIMER=RTC lib lapack_testing 2>&1 | tee MAKE-part2.log
cd TESTING
make -k TIMER=RTC 2>&1 | tee ../MAKE-testing.log
cd ..

Note the -k option to prevent the build to abort when it tries to run the generated Blue Gene binaries on the build system.

Testing the build.

The LAPACK test procedure requires end-of-file to be handled on stdin. This appears to not always work. For this reason we have modified the test routines to accept a magic input line (sentinel) to denote end-of-file. Modify the input files to have the sentinel in them

./fix-input-files

Submit the testing job (do not forget to edit the batch script to fit your site first). For BG/L runs there is a similar script llrun-bgl.sh.

cd TESTING
llsubmit llrun.sh

Inspect the output

../SummarizeRuns.bash
Errors were 371 out of 3161696.

None of the errors appears to us be alarming. They all relate to the level of accuracy in the computed answer. Raising the acceptance threshold in the input files makes the errors go away. In our experiments -O3 -qstrict gave similar accuracy to -O2. For this reason we chose to use -O3 -qstrict in the final file.

::::::::::::::
AllErrorsSummary-O2
::::::::::::::
Errors were 369 out of 3161696.
::::::::::::::
AllErrorsSummary-O3-qnostrict
::::::::::::::
Errors were 10857 out of 2126300.
::::::::::::::
AllErrorsSummary-O3-qstrict
::::::::::::::
Errors were 371 out of 3161696.


/gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=default
Errors were 371 out of 3161696.
/gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=100.0
Errors were 278 out of 3161696.
/gpfs/fs2/frontend-1/nsmeds/lapack/output-ratio=500.0

CBLAS

CBLAS

http://www.netlib.org/blas/blast-forum/
http://www.netlib.org/blas/blast-forum/cblas.tgz

Our collection of files for the build BlueGene-CBLAS-buildpack.tar.gz

Unpack the distribution

tar zxvf cblas.tgz
cd CBLAS

Download our BlueGene CBLAS build pack into the CBLAS directory. Unpack the file

tar -zxvf BlueGene-CBLAS-buildpack.tar.gz

The tar file will create a soft link to Makefile.BGP. If you are on a BG/L system redirect this link to point to Makefile.BGL instead.

Edit Makefile.in. CBDIR and BLLIB are typically needed to be modified. A good value for CBDIR is the root of the CBLAS source tree. BLLIB needs to be adjusted to match your installation of ESSL.

Patch the test source code to work around a possible stdin EOF problem on Blue Gene and also to turn off error code check tests in the test examples. This is because by default ESSL exits with an error message on erroneous input.

patch -p0 < cblas-testing-bgp.patch

Build the libcblas_bgp.a library

make all

Add the file sentinel that the patch file enables to input files

./fix-input-files

Run the tests

cd testing

Edit the submit script to match your site before submission. There is an alternative script for BG/L named llruntest-bgl.sh.

llsubmit llruntest.sh