Python with VirtualEnv

Managing several needed packages within a shared resource environment is difficult. Users may need different package versions, or even packages that conflict with each other. In this example, we will show you how to deal with virtualenv for python modules.

To setup virtual environment

We start by setting up a virtual environment on your home space

mkdir example4
virtualenv-3.6 --no-download ./example4/env

To use the virtual environment

cd example4
source ./example4/env/bin/activate.csh #This step depends on your SHELL 

Now, you are in a virtual environment. You may see [env] at the beginning of SHELL command. The following commands will tell you which python you are using:

which python
python --version

To play with the virtual environment, you can try with numpy which comes with the virtual environment. Here is an example of

import numpy as np

a = np.zeros((2,2))   # Create an array of all zeros
print(a)              # Prints "[[ 0.  0.]
                      #          [ 0.  0.]]"

b = np.ones((1,2))    # Create an array of all ones
print(b)              # Prints "[[ 1.  1.]]"

c = np.full((2,2), 7)  # Create a constant array
print(c)               # Prints "[[ 7.  7.]
                       #          [ 7.  7.]]"

d = np.eye(2)         # Create a 2x2 identity matrix
print(d)              # Prints "[[ 1.  0.]
                      #          [ 0.  1.]]"

e = np.random.random((2,2))  # Create an array filled with random values
print(e)                     # Might print "[[ 0.91940167  0.08143941]
                             #               [ 0.68744134  0.87236687]]"

To run


Note that, if your job cannot run outside the virtual environment if numpy is not available.

To install package

Here is an example of how to install ephem (python package for performing high-precision astronomy computations):

pip install --upgrade pip #do this at least for the first time that you use virtual environment
pip install --upgrade setuptools #do this at least for the first time that you use virtual environment
pip install ephem

Here is an example to use ephem to calculate the astrometric geocentric right ascension and declination for Mars on specific date.
import ephem
mars = ephem.Mars()

You then can simply run python in the virtual environment. You may see


To leave from the virtual environment


If you try to run again the previous example, but outside the virtual environment, you may see

Traceback (most recent call last):
  File "", line 1, in <module>
    import ephem
ImportError: No module named ephembecause there is no ephem install in the shared environment.

It is because we do not have ephem installed in the shared (central) environment.

Using VirtualEnv with Slurm

Here is an example (example4.slurm) of the Slurm submission script to run python application on worker node using VirtualEnv set in the frontend node (as the example above). To submit, you can simply use the same submission command sbatch example4.slurm

#SBATCH --qos=cu_hpc
#SBATCH --partition=cpu
#SBATCH --job-name=example4
#SBATCH --output=example4_log.txt
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:10:00
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G

module purge

#To handle PATHs
export MYCODEDIR=`pwd`

#To run C++ program
source $MYCODEDIR/example4/env/bin/activate

Last updated