Spack

Spack Introduction and Environment Setup

What is Spack?

Spack is a package manager, similar to yum, apt-get, or conda. It allows for multiple package versions/configurations. Unlike the other package managers listed, Spack is specifically designed to be used on HPC resources with varying architectures.

The purpose of this documentation is to provide a practical guide to quickly enable you to take advantage of its use in your research.

How to set up your environment to use Spack

First, log in to one of the gateway nodes; here, we are logging onto pascal:

If off campus, use this command:

Next, add the following code to your ~/.bashrc, which will automatically load Spack into your shell on all servers besides gateways:

Package Information

How to show what packages are installed in Spack

The Spack Basic Usage guide demonstrates how to show which packages are already present in Spack though we will show them here as well.

To see all packages currently installed, use this command:

Currently, there are 1027 packages (including different versions of the same package) installed. This number will undoubtably increase after the writing of this documentation.

Note: the name of each package is given, as well as the version number (e.g. autoconf@2.69). Also notice that the packages are also grouped by the OS and compiler that was used to construct the packages (e.g. linux-rhel6-x86_64 / gcc@4.4.6).

How to search the output of: spack find

However, while spack find list the packages alphabetically, it can still be a little difficult to search for packages you are interested in (especially if you don't know its exact/full name). For example, this command doesn't find any packages:

However, from the output of the command spack find clearly shows numerous packages that start with "py". To improve on Spack's native find command, add this to your ~/.bashrc:

This code provides the command spack_my_find, which provides a much more robust search of what's installed in Spack; specifically, it allows you to search for any package that contains a given string (regular expressions are supported). For example,

Produces the following output:

Getting information about installed packages

In the previous section, we observed that there were 2 installations of python@3.6.0 for rhel6 using gcc version 6.3.0. One way to see the differences between these different installations is the use the following command:

In the above command:

• The -v will cause the variants (e.g. compile-time options in package configuration) to be printed. This will allow us to see differences between the 2 packages.

• The -l will cause the package "hashes" to be printed.

Note: a hash is a fixed-length string that is calculated by a hash function, with the software/package of interest as the primary input. If 2 pieces of code have ANY differences, then they will have different hashes. Spack simply uses hashes to, among other things, provide a unique descriptor for each package.

The command produces the following output:

First, notice that the unique hashes (the first 7 characters) are given printed in grey, to the left of the each package name. These unique hashes will become important if you wish to use one of these 2 packages

Second, notice that each packages variants are printed with either a + or ~, indicating that an option is either included or not. For example, the first package (/u6dzm4v) does not include CUDA (~cuda), whereas the second package (/olcttts) does include CUDA (+cuda). Next, I'll describe how I learn more information about what the various package variants mean for a given package.

More package details!

In the previous section, we observed that packages can have different variants (e.g. compile-time options). However, spack find -l -v prints abbreviated variant names, which can sometimes be cryptic. For a more complete description of a packages potential variants (as  well as versions/dependencies), use this command (here, we're looking at Relion, but you could enter any package name):

OUTPUT:

This output is pretty self-explanatory. First, the name of the package is given, along with its home page. This is followed by a list of package versions that are currently considered stable by Spack. Next, a description of the specific package variants is given; defaults are shown in square brackets. This is

followed by installation information. Finally, package dependencies are described. These are automatically loaded when the package is loaded.

Listing available packages

Seeing what packages COULD be installed is also very easy! Just use the following command:

TRUNCATED OUTPUT:

Currently, there are 6,768 packages available! This number will also certainly increase after the writing of this documentation. 

Note: That the output of spack list is plain text, so searching its output is easier than searching the output of spack find

To search this output, just grep its output:

(where you can replace "^r-" with whatever regular expression you wish)

TRUNCATED OUTPUT:

Using Spack Packages

How to load Spack packages

Say we want to use an installation of python that is managed by Spack. First, let's see what versions/installations of python are currently installed:

OUTPUT:

Notice that there are multiple installations of python, that differ in version number, compiler, etc. As such, it is always a good idea to use the command spack find to observe if there are multiple installations before just loading a package blindly (otherwise you may load an installation that you do not intend).

To load the installation of python v. 3.7.0 that was compiled with GCC v. 6.3.0, use this command:

• The -r instructs Spack to load all modules that python@3.7.0 depends on. This is typically done automatically, but not currently done for perl, python, or R (this is because Spack does not build these packages with RPATH support--this may change in the future).

Executing this command doesn't produce any output, but just to illustrate that python@3.7.0 is indeed loaded (as well as its dependencies), execute the following command:

Here, we can see that python@3.7.0 is indeed loaded, as are all packages that it depends on.

How to use a unload package

To avoid confusion, as well as conserve memory, you may wish to unload a Spack package (this is especially true in complex workflows). Doing so is very easy:

Let's check and make sure we have really unloaded python@3.7.0:

OUTPUT:

Related articles