High Performance Computing Lesson 1

This is a hodge-podge collection of information, advice, and guidelines on what you should know:
  • All systems are different
    • Take advantage of system web pages for help.
    • Be particularly careful on HPC systems (they are an important resource).
  • Passwords
    • Find a system to store your passwords (paper, lastpass, dashlane, browser-based, etc.)
    • Do not use the same password twice
    • Do not share your passwords with anyone. If a sysadmin needs to access your account, they can do it without your password.
  • SSH
    • Make sure you know:
      • the difference between ssh, the ssh-agent, and an SSH keychain,
      • how to use ssh -l username host,
      • What ssh-add -L does,
      • how to create and use the files called "authorized_keys" and "config" in the .ssh directory,
      • and how to use sftp to transfer files over ssh.
    • Verbose mode, ssh -v is very helpful in debugging connection issues
    • Learn what ports common internet traffic occurs over (Some ISPs block some ports. Starbucks, for example, sometimes blocks ports for svn or ssh, depending on the particular location.)
    • See also my ssh, svn and git explainer.
    • To practice, try transferring a file using sftp from the HPC system to a local machine on campus which accepts incoming sftp transfers.
  • Backups
    • Keep a backup of everything.
  • Know how to use chmod to change file permissions
    • Basic usage is chmod [mode] [filename]
    • Use:
      • 644 (rw-r--r--) for public files
      • 600 (rw-------) for private files
      • 755 (rwxr-xr-x) for public directories and executables
      • 700 (rwx------) for private directories and executables
      • Eg. chmod -R u+rwX,go+rX,go-w <directory> is very useful for setting permissions recursively.
  • Filesystems
    • Make sure you know the difference between your user directory and the lustre or scratch filesystem and what each of those filesystems is useful for. Are they both backed up on your HPC system? Are old files purged?
    • Know how to determine your quota
    • Work plan:
      • Develop code and compile on on local machine
      • Recompile in user directory on HPC system
      • Copy executable and input files from user directory to lustre
        • I like to designate a special makefile target for this, e.g. "make lustre_copy"
      • Submit job from lustre
      • Copy output files from lustre to local machine using sftp
    • Optional: Create a lustre link with "ln -s [real location] [new alias]". My preferred location for the alias is e.g. ~/lustre .
  • SVN and git
    • Typically something like:
      • svn --username=[user] --password=[pass] checkout svn://hostname/directory
      • svn checkout svn+ssh://hostname/directory
      • git clone https://github.com/awsteiner/bamr
  • Compiling code
    • There is typically a compiling step, and a linking step
      • Compiling step: gcc -c -o test.o test.c
      • Linking step: gcc -o test test.o
    • In the compiling step, use -I flags to specify the location of include files, -W flags to change code warnings, and -O (Capital letter "o") flags to handle optimization
    • In the linking step, use -L flags to specify the location of libraries
    • There is a difference between linking with "shared" or "static" libraries. Using static libraries sometimes requires the flag "-static" during linking, and it combines all of the code into one large standalone executable. Without this flag, the executable contains only the base code, and when the code is actually executed, the program looks on the operating system (it typically looks in a few system directories and all the directories contained in the LD_LIBRARY_PATH variable) for the shared libraries and loads them in memory at the time of execution. Compilation with shared libraries is often easier on HPC systems, but static linking creates slightly faster code.
  • Compilers
    • Typical choices are gnu, Intel, PGI, etc.
    • Choice of compiler is a tradeoff between speed, features, and compatibility
    • I sometimes use modern C++ features not available in older compilers
  • Modules
    • Know how to use:
      • module show [module]
      • module load
      • module list
      • module apropos [search string]
    • Consider creating a text file (e.g. named module_script) which loads your favorite modules and then load them using "source module_script".
  • Environment variables and .bash_profile (or .profile)
    • Know how to set environment variables in your .bash_profile file (or the corresponding file for whatever shell you are using)
      • Often something like: "export VARIABLE=value"
    • Know how to get the current environment variables using env.
    • Good environment variables to know:
      • PATH: where shell looks for executables. I often add "." to PATH to avoid having to type "./" in front of executables.
      • LD_LIBRARY_PATH: where shell looks for shared libraries
      • EDITOR: default text editor (emacs, pico, vim, etc. Used, for example, by svn.)
      • HOST: sometimes set to host name
      • USER: sometimes set to user name
      • PWD: current working directory
      • HOME: home directory
      • CC: C compiler
      • FC: Fortran compiler
      • CXX: C++ compiler
      • LDFLAGS: Linker flags (for all compilers including C, C++, Fortran, etc.)
      • CFLAGS: C compiler flags
      • CXXFLAGS: C++ compiler flags
      • FFLAGS: Fortran compiler flags
        •  
      • MACHINE: What I use to distinguish between different machines
      • MPI_CXX: What I use to set the MPI C++ compiler
  • Makefiles
    • Get comfortable with targets, dependencies, and commands
    • Learn how environment variables can be used or overwritten inside makefiles.
    • Sometimes makefiles use environment variables or makefile variables to specify compiler flags. Either way, the notation in the makefile is the same: $(VARIABLE) is replaced with the value of the environment or makefile variable.
    • A common motif is to do something like 
      		  test.o: test.cpp
		  gcc -I$(SOME_INC) -o test.o -c test.cpp
      to specify that the value of the variable SOME_INC should be searched for header files. For example, if the value of SOME_INC was /home/username/include then the code above would be equivalent to 
      		  gcc -I/home/username/include -o test.o -c test.cpp
    • Some systems define environment variables which include the "-I" part. I don't recommend this because it is non-standard, but you should know how to handle it. If SOME_INC has the value "-I/home/username/include", then the correct makefile command would be 
      		  test.o: test.cpp
		  gcc $(SOME_INC) -o test.o -c test.cpp
    • Don't use make -j to compile with more than one processor on login nodes.
    • I use "ifeq ($(MACHINE),machinename)" to choose between machines
  • [UTK only] HW 1: Compile bamr by obtaining it from git clone https://github.com/awsteiner/bamr and update the makefile so it compiles on your HPC system.

Back to Andrew W. Steiner at the University of Tennessee.