Xmipp

Building With SCons

Scons is a python replacement for configure/automake etc.We have split the scons compilation into two different steps:

  1. The configure step: where one tells Scons about the relevant system setup. This step often requires user input.
  2. The compile step: where one actually compiles the whole package. This step is automated.

From Xmipp 3.1 on, the two steps are handled in setup.py (instead of scons.configure and scons.compile). As usual, for up-to-minute details you can look directly into the setup.py script. More info about setup.py in InstallationInternals

Configure step (Xmipp 3.1 and newer)

For configure step, setup.py calls scons with mode=configure:

xmipp_python external/scons/scons.py mode=configure -j PROCS --config=force OPTIONS

Relevant options:

(@todo: check this option table against current SConscript)

Option Default value Short explanation
CC gcc The C compiler
CXX g++ The C++ compiler
CXXFLAGS None The C++ compiler flags
LINKERFORPROGRAMS g++ Linker for programs
MPI_CC mpicc MPI C compiler
MPI_CXX mpiCC MPI C++ compiler
MPI_INCLUDE /usr/include MPI headers dir
MPI_LIB mpich MPI library
MPI_LIBDIR /usr/lib MPI libraries dir
MPI_LINKERFORPROGRAMS mpiCC MPI Linker for program
QTDIR /usr/share/qt3 Where is QT installed
QT_LIB qt-mt QT library to use
TIFF_INCLUDE /usr/include TIFF headers dir
TIFF_LIBDIR /usr/lib TIFF libraries dir
gui yes Build GUI (qt) programs?
gtest no Build tests?
tiff yes Build the Tiff programs?
mpi yes Build the MPI programs?
static no Prevent dynamic linking?

So, let's say compilation is failing because MPI cannot be found. Maybe the MPI you want is located in a place that the setup does not expect. You can help it with the MPI options, mainly MPI_INCLUDE and MPI_LIBDIR:

xmipp_python external/scons/scons.py mode=configure MPI_INCLUDE=/usr/lib/openmpi/include MPI_LIBDIR=/usr/lib/openmpi/lib

Environment variables are also useful. For instance, if you define JAVA_HOME, you will help scons to find jni.h

You can find more scons recipes in TheCNBCorner

Compile step (Xmipp 3.1 and newer)

For compile step, setup.py calls scons with mode=compile:

xmipp_python external/scons/scons.py mode=compile -j PROCS OPTIONS

You can instruct scons to build tests with gtest=1:

xmipp_python external/scons/scons.py mode=compile -j 4 gtest=1
# or, using setup.py
./setup.py -j 4  compile gtest=1

./scons.configure (before Xmipp 3.1)

How to use ./scons.configure

One has to tell SCons which compilers to use (compilation is most fluent using the GNU compilers, but the package has also been compiled using the intel (icc) or IBM (xlc) compilers. Apart from the compilers, one has to tell SCons where the external libraries are installed on the system. The Xmipp package depends mainly on 3 external libraries:

  1. tiff-devel: Only for the program xmipp_convert_tiff2raw
  2. qt3-devel: For programs displaying images, rotational spectra, micrographs etc. (e.g. xmipp_show, xmipp_micrograph_mark etc)
  3. mpi: For running parallel programs using the message passing interface.

To provide SCons with all this information, ./scons.configure has multiple arguments and corresponding default values. The relevant arguments for a normal installation are given in the table below. Dont worry about the other options, like MATLAB, QT4 or SWIG: they are not needed for a typical installation. Note that one may get the complete list of arguments by typing =./scons.compile --help".

Option Default value Short explanation
CC gcc The C compiler
CXX g++ The C++ compiler
CXXFLAGS None The C++ compiler flags
LINKERFORPROGRAMS g++ Linker for programs
MPI_CC mpicc MPI C compiler
MPI_CXX mpiCC MPI C++ compiler
MPI_INCLUDE /usr/include MPI headers dir
MPI_LIB mpich MPI library
MPI_LIBDIR /usr/lib MPI libraries dir
MPI_LINKERFORPROGRAMS mpiCC MPI Linker for program
QTDIR /usr/share/qt3 Where is QT installed
QT_LIB qt-mt QT library to use
TIFF_INCLUDE /usr/include TIFF headers dir
TIFF_LIBDIR /usr/lib TIFF libraries dir
gui yes Build GUI (qt) programs?
tiff yes Build the Tiff programs?
mpi yes Build the MPI programs?
static no Prevent dynamic linking?
swig no execute swig?

How to find your compilers and external libraries

(Part fo the following assumes your system uses Redhat package manager (rpm)).

The C/C++ compilers

We recommend using the GNU C/C++ compiler. Still, icc, and xlc have also been used successfully to compile Xmipp. In that case, make sure these compilers are in your PATH, and change the CC, CXX, and LINKERFORPROGRAMS arguments (e.g. CC = icc, and CXX = icc).

mpi

MPI is a Message Passing Interface. It is used by some Xmipp parallel programs to make it possible to distribute heavy CPU time consuming jobs load across different processors. There exist many implementations of the MPI library. Nevertheless, some of those implementations have shown to behave better than others because of many reasons like stability, memory issues or simply because they are more or less new.

If you are running on a system using a proprietary MPI implementation you will probably be doing well using it. If not, our test have shown that OPEN-MPI is a good and very stable alternative. OPEN-MPI is a recent, well-documented, easy-to-use, MPI implementation which was born from the union the of various independent MPI implementations. For more information on OPEN-MPI have a look at: http://www.open-mpi.org/

In practical terms, your MPI compilers are often in non-standard locations, although this depends on the particular MPI installation on your system. You may locate mpirun to get an idea where they are installed using:

which mpirun
And then look inside the corresponding directory structure:
  • for a bin directory. This directory should be in your PATH. If it is not add a line with (e.g.) export PATH = /usr/local/openmpi/bin to the .bash_profile file in your home directory. Inside this directory, look for executables like mpicc (usually there) and mpiCC. The latter is sometimes called mpicxx. If that is your case, add the argument MPICXX = mpicxx.
  • for a include directory, this directory will be your MPI_INCLUDE argument. If it is not the default, add the corresponding argument (e.g. MPI_INCLUDE = /usr/local/openmpi/include).
  • for a lib directory, which will your MPI_LIBDIR argument. Inside this directory look for a file called libmpi.so or libmpich.so. The latter is the default, if your library is called libmpi.so, add the argument MPI_LIB = mpi.

Alternatively, one may locate the installed MPI flavour, using:

rpm -qa|grep mpi
Look for openmpi, mpich, etc. And then use:
rpm -ql openmpi |less
to see where mpi is installed. Then look for the lib, include and bin directories as explained above.

qt3-devel

rpm -qa|grep qt3
Look for qt3-devel. If you don't have this, install this library or ask your system manager to do so. Now see where qt3-devel is installed:
rpm -ql qt3-devel|less
Look for something like = /usr/lib64/qt-3.3/=, and locate the lib directory therein. This will be your QTDIR argument for ./scons.configure. If this is different from the default, add the corresponding argument: (in this case) QTDIR =  /usr/lib64/qt-3.3/lib.

Inside the QTDIR, look how the library is called. There are two possibilities: libqt.so or libqt-mt.so. The latter is the defualt. If your library is called libqt.so, add the QT_LIB = qt argument to your configuration.

tiff-devel

See whether tiff-devel is installed on your system (it most often will be) using:

rpm -qa|grep tiff
Look for something with tiff-devel. If it is not there, ask your system administrator to install libtiff-devel. Then, see where it is installed using:
rpm -ql|grep libtiff-devel|less
Look for the following directories:
  • include: This wil be your TIFF_INCLUDE directory. Add the argument if it is not in the default (/usr/include) location.
  • lib: This will be your TILL_LIBDIR directory. Add the argument if it is not in the default (/usr/lib) location.

swig

Python modules may access c++ libraries through swig. If you are a developer add the flag swig=1 to your configure command otherwise the xmipp libraries and the swig library will not match. If you are a developer please install the swig module.

See whether swig is installed on your system (it most often will be) using:

rpm -qa|grep swig
Look for something with swig. If it is not there, ask your system administrator to install swig.

Running ./scons.configure

Now you will have a list of arguments to run ./scons.configure. The command you will execute in the Xmipp root directory may look like this (if you need more exmaples, see TheCNBCorner for a list of ./scons.configure commands that we use on our machines):

./scons.configure QTDIR=/usr/lib64/qt-3.3 MPI_LIBDIR=/usr/local/openmpi/lib MPI_LIB=mpi MPI_INCLUDE=/usr/local/openmpi/include TIFF_LIBDIR=/usr/lib64

If all goes well, you will get the following output.

** Use scons.configure --help to show a help message
scons: Reading SConscript files ...
* Configuring  ...
* QT3 selected!
* Trying environment's $QTDIR
Checking for QApplication qapp(0,0) in C++ library qt-mt... yes
Checking for TIFFGetVersion() in C library tiff... yes
* Checking for MPI ... yes
* Configuring FFTW (see build/fftw_configure.log for details) ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
Especially note the three yes when "Checking for .. QApplication, TIFF or MPI. If one of these is no, then SCons has not found the relevant libraries or executables. In that case, you may check the build/config.log to see what went wrong. You may still have to install some of the external libraries, or some of the arguments is still not correct.

Note that Xmipp will still compile if one or more of these Checks give no, but the corresponding programs (e.g. xmipp_show for qt, xmipp_convert_tiff2raw for tiff, or any xmipp_mpi_... for mpi) will just not be compiled.

Also not that Xmipp is being actively developed and we make a new (more powerful) release approximately once a year. The way of installing this release usually does not change, so you may want to save you configure command somewhere.

./scons.compile (before Xmipp 3.1)

Compilation is now (usually) quite smooth. Just use:

./scons.compile -j 4
Where the -j 4 option means the compilation will be done using 4 parallel threads. Use this if you have a multi-core machine. Now the hard work has been done. Just sit back and wait until Xmipp is compiled. After compilation, don't forget to set the PATH and LD_LIBRARY_PATH as explained in HowToInstall#Step_3_Edit_the_environment_setu.

Other stuff

To clean up:

  • Type ./scons.compile -c for cleaning
  • Type ./scons.compile -distclean to delete configuration also (still under testing)

To build just a single program or library:

  • Add the path
  • Or the final name (with xmipp for programs)

Other interesting flags:

  • -Q to suppress some messages
  • -s to suppress most messages
  • -c $TARGET_NAME to clean it
  • -n to just print what would be done