Peculiarities of the EM64T platform
Compiler issues
64-bit libraries are mostly kept in directories named lib64 instead of just lib.
For instance, the 64-bit version of the C-library is /lib64/libc.so.6
while the 32-bit version is /lib/libc.so.6.
The linker knows when a 64-bit appliction is being run and will pick the right kind of library for you. However, you cannot mix 32 and 64 bit objects in the same application so a potential cause of confusion is if you have old 32-bit object files or libraries lying around, as they can't be linked with a 64-bit application.
Another source of confusion is the fact
that lots of applications doesn't know about the lib64 convention.
Many of the applications and libraries installed in /pdc/vol keep
their libraries in lib, not lib64. All those sofware packages
have been compiled in 64-bit mode though. You must keep this in
mind when listing explicit link paths at compile time or in the
LD_LIBRARY_PATH variable. The simple rule
is that lib64 should be used if such a directory is present, otherwise
it is safe to use lib. Use module display on the module to see
the recommended paths.
All installed compilers default to 64-bit targets so you normally don't have to do anything but recompile.
32-bit binaries can be run without recompilation but they cannot use the all memory on the nodes. Only 64-bit binaries are supported by PDC and only 64-bit versions of dynamic libraries are installed. We encourage all users to recompile their codes on Lenngren.
While the AMD64 and EM64T architectures are binary compatible (it is possible to run code compiled for AMD Opteron or Athlon 64 processors on a Xeon with EM64T and vice versa), there are pitfalls. To get best performance code should be optimized for the EM64T processors. Code compiled for Opteron processors may not run as fast as it could on Xeons due to differences in caches and depth of the execution pipeline.
One actual incompatibility concerns multimedia extensions to the processors. AMD processors support 3Dnow! instructions while Xeons do not. Code that contains 3Dnow! instructions will not run on Xeon processors.
Linker errors with Intel compilers
The different memory models of the x86-64/EM64T binary format makes it necessary to use dynamic linked run-time modules with the Intel compiler when writing programs that will access large amounts of memory (>2GB). The flag -i_dynamic tells the compiler/linker to use the dynamic version of the Intel library functions. A typical indicator to use the -i_dynamic flag are messages like the following
libifcore.a(for_init.o)(.text+0x95): In function `for_rtl_init_': : relocation truncated to fit: R_X86_64_PC32 for__l_excpt_info
Using large amounts of memory
The EM64T processor adheres to the x86-64 ABI. This means that object files come in three flavours. Small, medium and large. Object files in the small memory model are restricted to 2GB of total address space (code and data). The medium memory model restricts the code section to reside within 2GB while data addressing is 64 bit. The large model allows both code and data to be larger than 2GB.
The Intel compiler as well as the Portland Group (PGI) compilers for x86-64 (EM64T) has an option -mcmodel={small,medium,large} to select which type of addressing should be generated in the object
file. Note that using large amount of static data may make it necessary to use the large memory model.
When creating dynamic libraries (.so) the resulting libraries contain position independent code (-fPIC) and can be used with any of the models. The runtime linker makes the correct translations for the executable.
When creating static libraries (.a) the release notes documentation states that the object files in the library should have the same memory model as the final executable. All program units should therefore be compiled with the same -mcmodel option.
In practice it appears as if you can mix object files of different models and the linker makes the necessary translations. However, in what cases this does not work and if the linker is able to diagnose these cases is not known. For this reason we advice to conform to the rule to not mix memory models between program units.
