Spack Introduction and Environment Setup
What is Spack?
Spack is a package manager, similar to yum, apt-get, or conda. It allows for multiple package versions/configurations.
Unlike the other package managers listed, Spack is specifically designed to be used on HPC resources with varying architectures.
The purpose of this documentation is to provide a practical guide to quickly enable you to take advantage of its use in your research.
How to set up your environment to use Spack
First, log in to one of the gateway nodes; here, we are logging onto pascal:
ssh username@pascal
If off campus, use this command:
ssh username@pascal.med.cornell.edu
Next, add the following code to your ~/.bashrc, which will automatically load Spack into your shell on all servers besides gateways:
# detect kernel version, then source spack file if it exists
_KERNEL_VERSION=$(uname -r | cut -d '.' -f1,2,3)
if [[ "${HOSTNAME}" != "pascal.med.cornell.edu" ]] || [[ "${HOSTNAME}" != "aphrodite.med.cornell.edu" ]] || [[ "${HOSTNAME}" != "aristotle.med.cornell.edu" ]]; then
if [[ "${_KERNEL_VERSION}" == *"2.6.3"* ]] ; then
if [ -f /softlib/apps/EL6/spack/share/spack/setup-env.sh ] ; then
. /softlib/apps/EL6/spack/share/spack/setup-env.sh
fi
fi
if [[ "${_KERNEL_VERSION}" == *"3.10"* ]] ; then
if [ -f /software/spack/centos7/share/spack/setup-env.sh ] ; then
. /software/spack/centos7/share/spack/setup-env.sh
elif [ -f /software/spack/share/spack/setup-env.sh ] ; then
. /software/spack/share/spack/setup-env.sh
fi
fi
fi
builtin unset _KERNEL_VERSION
As always, after editing your ~/.bashrc, execute the command source ~/.bashrc
Package Information
How to show what packages are installed in Spack
The Spack Basic Usage guide demonstrates how to show which packages are already present in Spack though we will show them here as well.
To see all packages currently installed, use this command:
spack find
==> 1027 installed packages. -- linux-centos7-x86_64 / gcc@4.8.5 ----------------------------- autoconf@2.69 fftw@3.3.7 libjpeg-turbo@1.5.90 openmpi@3.0.1 py-six@1.10.0 autoconf@2.69 fftw@3.3.7 libpciaccess@0.13.4 openmpi@3.0.1 python@2.7.13 autoconf@2.69 fftw@3.3.7 libpciaccess@0.13.5 openmpi@3.0.1 python@2.7.14 autoconf@2.69 fftw@3.3.8 libpng@1.6.27 openmpi@3.0.1 python@3.4.3 automake@1.15 fftw@3.3.8 libpng@1.6.29 openmpi@3.0.1 python@3.5.2 automake@1.16.1 fftw@3.3.8 libpthread-stubs@0.4 openmpi@3.0.1 python@3.5.2 automake@1.16.1 flex@2.6.3 libsigsegv@2.11 openmpi@3.0.1 python@3.6.0 binutils@2.28 fltk@1.3.3 libtiff@4.0.9 openmpi@3.1.0 readline@7.0 binutils@2.28 fltk@1.3.3 libtool@2.4.6 openssl@1 readline@7.0 bison@3.0.4 font-util@1.3.1 libx11@1.6.5 openssl@1 readline@7.0 boost@1.64.0 fontconfig@2.11.1 libxau@1.0.8 openssl@1.0.2k relion@2.0.3 bowtie2@2.3.1 fontconfig@2.12.1 libxcb@1.13 openssl@1.0.2k relion@2.0.3 bwa@0.7.15 freetype@2.7 libxdmcp@1.1.2 openssl@1.0.2n relion@2.1 bzip2@1.0.6 freetype@2.7.1 libxml2@2.9.4 pcre@8.40 relion@2.1 cairo@1.14.8 gcc@6.3.0 libxml2@2.9.4 perl@5.24.1 relion@2.1 cairo@1.14.8 gdbm@1.13 libxml2@2.9.8 perl@5.26.2 relion@2.1 cmake@3.7.2 gdbm@1.14.1 libxslt@1.1.28 perl-text-soundex@3.05 relion@2.1 cmake@3.8.0 gettext@0.19.8.1 libxslt@1.1.29 pixman@0.34.0 relion@2.1 cmake@3.9.4 glib@2.49.7 libxslt@1.1.29 pixman@0.34.0 relion@2.1 cmake@3.9.4 glib@2.53.1 lz4@1.8.1.2 pkg-config@0.29.2 relion@3.0_beta cmake@3.9.4 gmp@6.1.2 lzma@4.32.7 pkgconf@1.3.10 relion@3.0_beta cmake@3.10.0 help2man@1.47.4 lzo@2.09 pkgconf@1.4.2 relion@3.0_beta cmake@3.11.2 hmmer@3.1b2 m4@1.4.18 py-appdirs@1.4.3 relion@3.0_beta
This command will not work on the gateway nodes, as Spack is not meant to be used on these nodes.
We have 2 instances of Spack running, for CentOS 6 and CentOS 7 accordingly. This means it is possible you may see some packages that are available when working on a CenOS 7 node will not be available on a CentOS 6 node, and vice versa. If you run into this, please let us know: scu@med.cornell.edu
Currently, there are 1027 packages (including different versions of the same package) installed. This number will undoubtably increase after the writing of this documentation.
Note: the name of each package is given, as well as the version number (e.g. autoconf@2.69). Also notice that the packages are also grouped by the OS and compiler that was used to construct the packages (e.g. linux-rhel6-x86_64 / gcc@4.4.6).
How to search the output of: spack find
However, while spack find list the packages alphabetically, it can still be a little difficult to search for packages you are interested in (especially if you don't know its exact/full name). For example, this command doesn't find any packages:
spack find py
However, from the output of the command spack find clearly shows numerous packages that start with "py". To improve on Spack's native find command, add this to your ~/.bashrc:
function spack_my_find () {
spack find | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | grep -E "${1}|-- linux" | sed 's/-- linux/\n-- linux/g'
}
As always, after editing your ~/.bashrc, execute the command source ~/.bashrc.
This code provides the command spack_my_find, which provides a much more robust search of what's installed in Spack; specifically, it allows you to search for any package that contains a given string (regular expressions are supported). For example,
spack_my_find beta
Produces the following output:
-- linux-centos7-x86_64 / gcc@4.8.5 ----------------------------- relion@3.0_beta relion@3.0_beta relion@3.0_beta relion@3.0_beta -- linux-centos7-x86_64 / gcc@6.3.0 -----------------------------
Note: Sometimes, you'll see multiple installations of the same package installed on the same OS, with the same compiler (e.g. relion@3.0_beta is listed 3 times in the above output). What this means is that this packages was installed for the same OS/compiler, but with different configuration options. This will be explained in the next section.
Getting information about installed packages
In the previous section, we observed that there were 2 installations of python@3.6.0 for rhel6 using gcc version 6.3.0. One way to see the differences between these different installations is the use the following command:
spack find -v -l relion@3.0_beta
In the above command:
- The
-vwill cause the variants (e.g. compile-time options in package configuration) to be printed. This will allow us to see differences between the 2 packages. - The
-lwill cause the package "hashes" to be printed.
Note: a hash is a fixed-length string that is calculated by a hash function, with the software/package of interest as the primary input. If 2 pieces of code have ANY differences, then they will have different hashes. Spack simply uses hashes to, among other things, provide a unique descriptor for each package.
The command produces the following output:
==> 4 installed packages. -- linux-centos7-x86_64 / gcc@4.8.5 ----------------------------- u6dzm4v relion@3.0_beta build_type=RelWithDebInfo ~cuda cuda_arch= +double~double-gpu+gui purpose=cluster olcttts relion@3.0_beta build_type=RelWithDebInfo +cuda cuda_arch=60 +double~double-gpu+gui purpose=cluster f2fjevh relion@3.0_beta build_type=RelWithDebInfo +cuda cuda_arch=60 +double~double-gpu+gui purpose=desktop opdhbcs relion@3.0_beta build_type=RelWithDebInfo +cuda cuda_arch=70 +double~double-gpu+gui purpose=cluster
First, notice that the unique hashes (the first 7 characters) are given printed in grey, to the left of the each package name. These unique hashes will become important if you wish to use one of these 2 packages
Second, notice that each packages variants are printed with either a + or , indicating that an option is either included or not. For example, the first package (/u6dzm4v) does not include CUDA (~~cuda), whereas the second package (/olcttts) does include CUDA (+cuda). Next, I'll describe how I learn more information about what the various package variants mean for a given package.
More package details!
In the previous section, we observed that packages can have different variants (e.g. compile-time options). However, spack find -l -v prints abbreviated variant names, which can sometimes be cryptic. For a more complete description of a packages potential variants (as well as versions/dependencies), use this command (here, we're looking at Relion, but you could enter any package name):
spack info relion
OUTPUT:
CMakePackage: relion
Description:
RELION (for REgularised LIkelihood OptimisatioN, pronounce rely-on) is a
stand-alone computer program that employs an empirical Bayesian approach
to refinement of (multiple) 3D reconstructions or 2D class averages in
electron cryo-microscopy (cryo-EM).
Homepage: http://http://www2.mrc-lmb.cam.ac.uk/relion
Tags:
None
Preferred version:
2.1 [git] https://github.com/3dem/relion.git
Safe versions:
develop [git] https://github.com/3dem/relion.git
3.0_beta [git] https://bitbucket.org/scheres/relion-3.0_beta.git
2.1 [git] https://github.com/3dem/relion.git
2.0.3 [git] https://github.com/3dem/relion.git
Variants:
Name [Default] Allowed values Description
build_type [RelWithDebInfo] Debug, Release, The build type to build
RelWithDebInfo,
Profiling,
Benchmarking
cuda [on] True, False enable compute on gpu
cuda_arch [] 20, 30, 32, 35, 50, CUDA architecture
52, 53, 60, 61, 62,
70
double [on] True, False double precision (cpu) code
double-gpu [off] True, False double precision gpu
gui [on] True, False build the gui
purpose [cluster] cluster, desktop build relion for use in
cluster or desktop
Installation Phases:
cmake build install
Build Dependencies:
cmake cuda fftw fltk libtiff mpi
Link Dependencies:
cuda fftw fltk libtiff mpi
Run Dependencies:
None
Virtual Packages:
None
This output is pretty self-explanatory. First, the name of the package is given, along with its home page. This is followed by a list of package versions that are currently considered stable by Spack. Next, a description of the specific package variants is given; defaults are shown in square brackets. This is
followed by installation information. Finally, package dependencies are described. These are automatically loaded when the package is loaded.
Listing available packages
Seeing what packages COULD be installed is also very easy! Just use the following command:
spack list
TRUNCATED OUTPUT:
==> 2831 packages. abinit muster r-catools abyss mvapich2 r-cdcfluview accfft mxml r-cellranger ack mxnet r-cghflasso activeharmony nag r-checkmate adept-utils nalu r-checkpoint adios nalu-wind r-chemometrics adios2 namd r-chron adlbx nano r-circlize adol-c nanoflann r-class aegean nanopb r-classint albany nasm r-cli
Currently, there are 2,831 packages available! This number will also certainly increase after the writing of this documentation.
Note: That the output of spack list is plain text, so searching its output is easier than searching the output of spack find
To search this output, just grep its output:
spack list | grep "^r-"
(where you can replace "^r-" with whatever regular expression you wish)
TRUNCATED OUTPUT:
r-a4 r-a4base r-a4classif r-a4core r-a4preproc r-a4reporting r-abadata
Using Spack Packages
How to load Spack packages
Say we want to use an installation of python that is managed by Spack. First, let's see what versions/installations of python are currently installed:
spack find python
OUTPUT:
==> 19 installed packages. -- linux-centos7-x86_64 / gcc@4.8.5 ----------------------------- python@2.7.13 python@2.7.14 python@3.4.3 python@3.5.2 python@3.5.2 python@3.6.0 -- linux-centos7-x86_64 / gcc@6.3.0 ----------------------------- python@2.7.13 python@2.7.14 python@2.7.14 python@2.7.15 python@3.4.3 python@3.6.0 python@3.7.0 python@2.7.13 python@2.7.14 python@2.7.15 python@2.7.15 python@3.5.2 python@3.6.2
Notice that there are multiple installations of python, that differ in version number, compiler, etc. As such, it is always a good idea to use the command spack find to observe if there are multiple installations before just loading a package blindly (otherwise you may load an installation that you do not intend).
Before loading any Spack packages, it is helpful to know that Spack stores a list of loaded packages in the variable $LOADEDMODULES. Executing the command echo $LOADEDMODULES at this point will produce a blank line, as we haven't loaded Spack packages yet (try it out, just to prove this to yourself).
To load the installation of python v. 3.7.0 that was compiled with GCC v. 6.3.0, use this command:
spack load -r python@3.7.0^gcc@6.3.0
- The -r instructs Spack to load all modules that python@3.7.0 depends on. This is typically done automatically, but not currently done for perl, python, or R (this is because Spack does not build these packages with RPATH support--this may change in the future).
Executing this command doesn't produce any output, but just to illustrate that python@3.7.0 is indeed loaded (as well as its dependencies), execute the following command:
echo $LOADEDMODULES | sed "s/:/\n/g" | sort
Here, we can see that python@3.7.0 is indeed loaded, as are all packages that it depends on.
How to use a unload package
To avoid confusion, as well as conserve memory, you may wish to unload a Spack package (this is especially true in complex workflows). Doing so is very easy:
spack unload python@3.7.0
Let's check and make sure we have really unloaded python@3.7.0:
echo $LOADEDMODULES | sed "s/:/\n/g" | sort
OUTPUT:
Related articles