Programming Environment on NCSA Intel 64 Linux Cluster

1. Compilers
1. Compiler Commands
2. Vendor Documentation
3. Useful Compiler/Linker Options
2. Libraries and Application Software
1. Intel's Math Kernel Library (MKL)
2. Locally Installed Software
3. Hierarchical Data Format (HDF) Library

1. Compilers

NCSA supports the Intel compilers on the NCSA Intel 64 Linux system. The current default is version 10.0. Version 9 is available via a SoftEnv key. The GNU compilers are also available.

1.1 Compiler Commands

Serial

To compile and link a serial program in Fortran, C, and C++ enter:

  ifort myprog.f
  icc myprog.c
  icpc myprog.cc

MPI

There are 3 MPI implementations available on the system: MVAPICH2, OpenMPI, and VMI2.

As of June 29 2007, the default implementation if you specify none is MVAPICH2.

You may explicitly add one of the other available MPI implementations to your environment via the soft add command to use the compilers shown. For running jobs, add the key [+keyname] to your $HOME/.soft file. Your .soft file should contain at most one MPI key at a time and it should be listed before any other keys.

Fortran 77: mpif77 myprog.f
Fortran 90: mpif90 myprog.f90
    C    : mpicc myprog.c
    C++    : mpicxx myprog.cc

Notes on MPI implementations:

• Making calls to system(), fork(), or popen() are not supported on Abe with Infiniband. This is due to a software issue with the InfiniBand drivers for MPI. It affects all the MPI implementations on Abe.
  [Note: You can call system()/fork()/popen() before the call to MPI_INIT or after the call to MPI_FINALIZE.]

  The problem can be difficult to diagnose since it may not occur for all runs. Also, the failure is not immediate after a call to system()/fork()/popen(), and the error messages generated can be very different between runs and not very meaningful. If you are unsure if the error messages you get are due to this, you can use the find command to look for instances of system(), fork(), and popen() calls in your source files:

  find . \( -name \*.c\* -or -name \*.f\* -or -name \*.C\* -or -name \*.F\* \) \
     -exec egrep --with-filename -i 'system *\(|fork *\(|popen *\(' {} \; -print
  

  Some workarounds:
  • If the calls are not critical, comment them out.
  • As mentioned previously, make the calls before the call to MPI_INIT or after the call to MPI_FINALIZE if possible.
  • Some commands such as mkdir, rmdir, chmod, and unlink have library call equivalents - replace with these (for Fortran codes, call the C routine from Fortran).
  • If you are using VMI2 and OpenMPI, you can also try TCP/IP over GigE but performance is going to suffer.

    For VMI2: mpirun -specfile tcp ...
    For OpenMPI: mpirun -mca btl self,sm,tcp -mca btl_tcp_exclude eth0,ib0 ...

• Open MPI and MVAPICH2 are MPI-2 standards compliant. However, VMI2 does not have all the APIs as specified in the MPI-2 standard. The only MPI-2 features that are present in VMI2 (built on top of mpich-1.2.x from Argonne) are the MPI-IO APIs. It is important to note that one sided MPI_Get and MPI_Put operations are not supported by VMI2.
• We recommend using the --disable-profiling switch when running VMI2 jobs.
• If you use a non-default implementation of MPI, for simplicity we suggest that you put the MPI soft key in your $HOME/.soft when you run jobs. The OpenMPI implementation requires it to be there (it doesn't work either if it is in the environment when you submit the job, or in the batch script).
• When using +mpichvmi-intel-ofed1.2 with C++ programs that call MPI::Init(), the arguments from main() must be passed:

    MPI::Init(argc, argv);
  

  Otherwise, the program will generate a segmentation fault runtime error at mpi startup.
• VMI2's mpirun doesn't like redirecting stdin, so use the option -stdin as in:

  mpirun -np 4 -machinefile $PBS_NODEFILE -stdin input.file ./a.out
  

OpenMP

To compile and link an OpenMP program in Fortran and C, use the -openmp option:

  ifort -openmp myprog.f
  icc -openmp myprog.c

Hybrid MPI/OpenMP

To compile and link an MPI/OpenMP hybrid program, use the -openmp option with the MPI compling commands:

  mpif77 -openmp myprog.f
  mpif90 -openmp myprog.f90
  mpicc -openmp myprog.c
  mpicxx -openmp myprog.cc

1.2 Vendor Documentation

• Intel Fortran compiler (more at Intel's web site)
• Intel Fortran compiler 10.0 Release Notes
• Intel C/C++ compiler (more at Intel's web site)
• Intel C/C++ compiler 10.0 Release Notes

1.3 Useful Compiler/Linker Options

Compatibility Options

-fpp[n]

runs the Fortran preprocessor on source files prior to compilation.

-assume byterecl

Many compilers assume that in the OPEN statement, the record length (recl=) of direct-access unformatted files is specified in bytes. The default is in four-byte words. If you use the option above, it assumes the record length is specified in bytes. (default is -assume nobyterecl)

-Dname[=value]

specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).

-w90, -w95

suppress messages about use of non-standard Fortran

-i8

set default KIND of integer variables is 8

-integer_size size

specifies the default size of integer and logical variables (size: 16, 32, 64)

-r8

set default size of REAL to 8 bytes

-save

save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g

produce symbolic debug information in object file (implies -O0)

-CB

array bounds checking

-fpen

specifies behavior on floating point exceptions (n = 0, 1 or 3)

-zero

implicitly initialize all data to zero

-u

equivalent to having "IMPLICIT NONE" throughout the code

-traceback

allow the display of source file traceback information at runtime when a severe error occurs.

Optimization

-O2

enable optimizations (DEFAULT)

-O1

optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit

-O3

enable -O2 plus more aggressive optimizations that may not improve performance for all programs

-O0

disable optimizations

-O

same as -O2

-ipo

enable multi-file IP optimizations (between files)

-mp

maintain floating point precision (disables some optimizations)

-mp1

improve floating-point precision (speed impact is less than -mp)

-xT

optimizes for the Intel Core2 Quad processor family

Shared Memory Parallel Programming Options

-openmp

enable the compiler to generate multi-threaded code based on OpenMP directives.

-openmp-report{0|1|2}

control the OpenMP parallelizer diagnostic level

Other Options

-v

display verbose information about each compiler phase.

-V

Displays the compiler version information

-sox

Tells the compiler to save the compiler options and version number in the executable. The default is -no-sox. To retrieve the information:

        strings --all a.out | grep comment:

-Wno-deprecated

Disable printing the warnings of deprecated flags/switches. To see deprecated options:

	icc -help deprecated
	ifort -help deprecated

Notes

• The Intel 10 compiler enables automatic vectorization using Intel SSE instructions (SSE|SSE2|SSE3) at optimization levels O2 and O3 without explicit invocation by the user via the -x<arch> switch. You can also add the use of SSSE3 (Supplemental Streaming SIMD Extensions 3) by using the -xT switch at compilation. The -xT switch should be treated as an additional optimization flag.
• You should use -shared-intel to specify the dynamic Intel libraries if you are linking against shared objects built by Intel compilers. A side effect of this change is that users may see the following message from the linker:

     warning: feupdateenv is not implemented and will always fail 
  

  This warning is due to a mismatch of library types and can be ignored. The warning will not appear if -shared-intel is used.
• If the static data size in your Fortran code is larger than 2GB, you may get relocation compiling error. Using -shared-intel option can solve the problem.

2. Libraries and Application Software

2.1 Intel Math Kernel Library

The Math Kernel Library (MKL) contains the complete set of functions from the basic linear algebra subprograms (BLAS), the extended BLAS (sparse), and the complete set of LAPACK routines. In addition, there is a set of fast Fourier transforms in single- and double-precision, real and complex data types with both Fortran and C interfaces. The library also includes the cblas interfaces, which allow the C programmer to access all the functionality of the BLAS without considering C-Fortran issues.

MKL is available via a SoftEnv key:

         soft add +intel-mkl

For blas: -L${MKL_HOME}/lib/em64t -lmkl -lguide -lpthread
For lapack: -L${MKL_HOME}/lib/em64t -lmkl_lapack -lmkl -lguide -lpthread
For scalapack: Please see the Intel MKL 10 software database entry for use of Intel MKL ScaLAPACK , BLACS and much more.

Note: Some MKL routines can run in parallel (using OpenMP) if the environment variable $OMP_NUM_THREADS is set to the number of threads. To ensure only one thread is being used in MKL routines set the value of $MKL_SERIAL to YES (this is the default).

• Intel Math Kernel Library Information
• Intel Math Kernel Library Quick Reference

2.2 Locally Installed Software

2.3 Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF (4.x and previous releases) and HDF5. These formats are completely different and NOT compatible.

HDF/HDF5 is available on NCSA HPC systems via the TeraGrid Coordinated TeraGrid Software and Services (CTSS). Use SoftEnv for information on accessing the software. Information on support is available at the HDF Support Issues page.

The HDF Home Page has detailed information on HDF and HDF5, including documentation, tutorials, examples, and FAQs.