Python with VirtualEnv

Managing several needed packages within a shared resource environment is difficult. Users may need different package versions, or even packages that conflict with each other. In this example, we will show you how to deal with virtualenv for python modules.

To setup virtual environment

We start by setting up a virtual environment on your home space

mkdir example4
virtualenv-3.6 --no-download ./example4/env

To use the virtual environment

cd example4
source ./example4/env/bin/activate.csh #This step depends on your SHELL 

Now, you are in a virtual environment. You may see [env] at the beginning of SHELL command. The following commands will tell you which python you are using:

which python
python --version

To play with the virtual environment, you can try with numpy which comes with the virtual environment. Here is an example of numpy_example.py

import numpy as np

a = np.zeros((2,2))   # Create an array of all zeros
print(a)              # Prints "[[ 0.  0.]
                      #          [ 0.  0.]]"

b = np.ones((1,2))    # Create an array of all ones
print(b)              # Prints "[[ 1.  1.]]"

c = np.full((2,2), 7)  # Create a constant array
print(c)               # Prints "[[ 7.  7.]
                       #          [ 7.  7.]]"

d = np.eye(2)         # Create a 2x2 identity matrix
print(d)              # Prints "[[ 1.  0.]
                      #          [ 0.  1.]]"

e = np.random.random((2,2))  # Create an array filled with random values
print(e)                     # Might print "[[ 0.91940167  0.08143941]
                             #               [ 0.68744134  0.87236687]]"

To run

python numpy_example.py

Note that, if your job cannot run outside the virtual environment if numpy is not available.

To install package

Here is an example of how to install ephem (python package for performing high-precision astronomy computations):

pip install --upgrade pip #do this at least for the first time that you use virtual environment
pip install --upgrade setuptools #do this at least for the first time that you use virtual environment
pip install ephem

Here is an example to use ephem to calculate the astrometric geocentric right ascension and declination for Mars on specific date.

#ephem_job.py
import ephem
mars = ephem.Mars()
mars.compute('2021/5/19')
print(mars.ra)
print(mars.dec)

You then can simply run python ephem_job.py in the virtual environment. You may see

7:08:08.24
23:55:30.6

To leave from the virtual environment

deactivate

If you try to run again the previous example, but outside the virtual environment, you may see

Traceback (most recent call last):
  File "ephem_job.py", line 1, in <module>
    import ephem
ImportError: No module named ephembecause there is no ephem install in the shared environment.

It is because we do not have ephem installed in the shared (central) environment.

Using VirtualEnv with Slurm

Here is an example (example4.slurm) of the Slurm submission script to run python application on worker node using VirtualEnv set in the frontend node (as the example above). To submit, you can simply use the same submission command sbatch example4.slurm

#!/bin/bash
#
#SBATCH --qos=cu_hpc
#SBATCH --partition=cpu
#SBATCH --job-name=example4
#SBATCH --output=example4_log.txt
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G

module purge

#To handle PATHs
export MYCODEDIR=`pwd`
echo "MYCODEDIR = "$MYCODEDIR
echo "TMPDIR = "$TMPDIR

#To run C++ program
cd $TMPDIR
cp $MYCODEDIR/ephem_job.py .
source $MYCODEDIR/example4/env/bin/activate
python ephem_job.py
deactivate