High Performance Computing Lesson 1
This is a hodge-podge collection of information, advice, and
guidelines on what you should know:
- All systems are different
- Take advantage of system web pages for help.
- Be particularly careful on HPC systems (they are an
- Find a system to store your passwords (paper,
lastpass, dashlane, browser-based, etc.)
- Do not use the same password twice
- Do not share your passwords with anyone. If a sysadmin
needs to access your account, they can do it without your
- Make sure you know:
- the difference between ssh, the ssh-agent, and
an SSH keychain,
- how to use ssh -l username host,
- What ssh-add -L does,
- how to create and use the files called
and "config" in the .ssh directory,
- and how to
use sftp to transfer files over ssh.
- Verbose mode, ssh -v is very helpful in
debugging connection issues
- Learn what ports common internet traffic occurs over
(Some ISPs block some ports. Starbucks, for example,
sometimes blocks ports for svn or ssh, depending on
the particular location.)
- See also my
svn and git explainer.
- To practice, try transferring a file using sftp
from the HPC system to a local machine on campus which
accepts incoming sftp transfers.
- Keep a backup of everything.
- Know how to use chmod to change file permissions
- Basic usage is chmod [mode] [filename]
- 644 (rw-r--r--) for public files
- 600 (rw-------) for private files
- 755 (rwxr-xr-x) for public directories and
- 700 (rwx------) for private directories and
- Eg. chmod -R u+rwX,go+rX,go-w <directory>
is very useful for setting permissions recursively.
- Make sure you know the difference between your user
directory and the lustre or scratch filesystem and what each
of those filesystems is useful for. Are they both backed up
on your HPC system? Are old files purged?
- Know how to determine your quota
- Work plan:
- Develop code and compile on on local machine
- Recompile in user directory on HPC system
- Copy executable and input files from user directory
- I like to designate a special makefile target for
this, e.g. "make lustre_copy"
- Submit job from lustre
- Copy output files from lustre to local machine
- Optional: Create a lustre link with "ln -s
[real location] [new
alias]". My preferred location for the alias
is e.g. ~/lustre .
- SVN and git
- Typically something like:
- svn --username=[user] --password=[pass]
- svn checkout
- git clone
- Compiling code
- There is typically a compiling step, and a linking
- Compiling step: gcc -c -o test.o test.c
- Linking step: gcc -o test test.o
- In the compiling step, use -I flags to specify the
location of include files, -W flags to change
code warnings, and -O (Capital letter "o")
flags to handle optimization
- In the linking step, use -L flags to specify
the location of libraries
- There is a difference between linking with "shared" or
"static" libraries. Using static libraries sometimes
requires the flag "-static" during linking, and it combines
all of the code into one large standalone executable.
Without this flag, the executable contains only the base
code, and when the code is actually executed, the program
looks on the operating system (it typically looks in a few
system directories and all the directories contained in the
LD_LIBRARY_PATH variable) for the shared libraries
and loads them in memory at the time of execution.
Compilation with shared libraries is often easier on HPC
systems, but static linking creates slightly faster code.
- Typical choices are gnu, Intel, PGI, etc.
- Choice of compiler is a tradeoff between speed, features,
- I sometimes use modern C++ features not available in
- Know how to use:
- module show [module]
- module load
- module list
- module apropos [search string]
- Consider creating a text file (e.g.
named module_script) which loads your favorite
modules and then load them using "source
- Environment variables and .bash_profile
- Know how to set environment variables in your
(or the corresponding file for whatever shell you are
- Often something like: "export VARIABLE=value"
- Know how to get the current environment variables using
- Good environment variables to know:
- PATH: where shell looks for executables. I often add
"." to PATH to avoid having to type "./" in front of
- LD_LIBRARY_PATH: where shell looks for shared
- EDITOR: default text editor (emacs, pico, vim, etc.
Used, for example, by svn.)
- HOST: sometimes set to host name
- USER: sometimes set to user name
- PWD: current working directory
- HOME: home directory
- CC: C compiler
- FC: Fortran compiler
- CXX: C++ compiler
- LDFLAGS: Linker flags (for all compilers including
C, C++, Fortran, etc.)
- CFLAGS: C compiler flags
- CXXFLAGS: C++ compiler flags
- FFLAGS: Fortran compiler flags
- MACHINE: What I use to distinguish between different
- MPI_CXX: What I use to set the MPI C++ compiler
- Get comfortable with targets, dependencies, and commands
- Learn how environment variables can be used or overwritten
- Sometimes makefiles use environment variables or
makefile variables to specify compiler flags. Either
way, the notation in the makefile is the same:
$(VARIABLE) is replaced with the value of the
environment or makefile variable.
- A common motif is to do something like
gcc -I$(SOME_INC) -o test.o -c test.cpp
to specify that the value of the variable
SOME_INC should be
searched for header files. For example, if the value
of SOME_INC was /home/username/include then
the code above would be equivalent to
gcc -I/home/username/include -o test.o -c test.cpp
- Some systems define environment variables which
include the "-I" part. I don't recommend this because
it is non-standard, but you should know how to handle it.
If SOME_INC has the value
"-I/home/username/include", then the correct
makefile command would be
gcc $(SOME_INC) -o test.o -c test.cpp
- Don't use make -j to compile with more than one
processor on login nodes.
- I use "ifeq ($(MACHINE),machinename)"
to choose between machines
- [UTK only] HW 1: Compile bamr by obtaining it from
https://github.com/awsteiner/bamr and update the
makefile so it compiles on your HPC system.
Back to Andrew W. Steiner at
the University of Tennessee.