Access to the ChargerNet File System is available for those users who need to access their home directory or project area from a remote client (on campus or via the campus VPN).

URC maintains a Samba server which allows SMB capable clients (i.e. MS-Windows, MacOS X, and Linux) to mount remote shares as a local drive or mount point.  To obtain permission to use this server, send email to URCSupport@uncc.edu.

Below are instructions for mounting a ChargerNet Samba Share from  various client machine types. Note that the Samba server  authenticates using Campus (NinerNet) passwords.

The examples below assume the following for ChargerNet user “joeuser”:

ChargerNet Samba Server:                        samba.urc.uncc.edu

ChargerNet login:                                         joeuser   (its.uncc.edu/joeuser)*

ChargerNet home directory:                      /users/joeuser

ChargerNet share name:                            joeuser

[ MS-Windows XP ]

1. Select ‘My Computer’

2. Select ‘Tools – > Map Network Drive’

3. Fill in path to folder (share)  i.e.  \\samba.urc.uncc.edu\joeuser

4. Select ‘Connect using a different user name’

5. Fill in username* (see footnote below) i.e    its.uncc.edu\joeuser

6. Fill in NinerNet password.

[ MacOS X ]

1. In ‘Finder’ application select Go -> Connect to Sever

2. Fill in Server address (and share) indicating smb protocol i.e. smb://samba.urc.uncc.edu/joeuser

3. Press ‘Connect’

4. Fill in Name (ChargerNet login)  i.e. joeuser (domain not required)

5. Fill in NinerNet password

[ Red Hat Linux - File Browser ]

1. In ‘File Broswer‘ application select File -> Connect to Server

2. Fill in the following:

Service Type: Windows share

Server: samba.urc.uncc.edu

Share: joeuser

Domain Name: its.uncc.edu

Name to use for connection: joeuser

3. Press ‘Connect’ button

4, Fill in NinerNet password.

[ Red hat Linux - Command Line ]

You can use the mount.cifs(8) command which is part of the samba(7) suite.  See the man page for mount.cifs for more details

To mount a share (to /home/joeuser/samba on the client) use the following command:

$ /sbin/mount.cifs //samba.urc.uncc.edu/joeuser  /home/joeuser/samba -o user=joeuser

To unmount the share:

$/sbin/umount.cifs /home/joeuser/samba

* Objects on the network including servers and even users are distinguished using “domain” naming conventions.  For instance, the samba server referenced above belongs to URC and is therefore part of the ‘urc’ domain.  Hence the full name of that object is: samba.urc.uncc.edu.

However, since the samba server uses the campus NinetNet system to authenticate user logins, the full user name (where required as in the case of a MS-Windows client)  uses the ‘its’ domain not the ‘urc’ domain.  Hence, a fully qualified user name is specified as:  its.uncc.edu\joeuser.

We are planning a  URC Webserver/Portal outage for Tuesday, September 8, 2009 from 7am until 8am.   We will use this outage to upgrade the webserver to Redhat EL 5.3.The following websites will be unavailable for the duration of the upgrade:

The URC Website:
http://www.urc.uncc.edu/urc/

The ChargerNet Portal:
https://chargernet.uncc.edu/portal/

The outage will impact access to the cluster through the ChargerNet Portal, but it will NOT impact cluster usage through the submit hosts.

This is a reminder that we are transitioning away from the Condor Job Scheduler to the Torque/Maui Scheduling system.  We have been running both systems for most of the summer in order to allow sufficient time for you to learn the new commands and update your submission scripts and procedures.  We have now moved most of our compute nodes into Torque/Maui and we plan to complete the transition and retire Condor on September 15th.

If you have not already done so, please begin submitting your cluster jobs using the new system as described on our web pages at http://www.urc.uncc.edu/urc/faqs/job-scheduling-with-torque/.  We have provided example scripts for our most popular applications (see folder /apps/torque/examples on the cluster).  The URC staff is also available to assist you with the transition if needed – simply send an email request to URCSupport@uncc.edu.

Application versions to be updated on the URC Cluster:

*ABAQUS 6.8-4
path = /apps/pkg/abaqus/6.8-4
env = /apps/usr/env/abaqus-6.8-4
*CFDACE 2009.0.14
path = /apps/pkg/cfdace-2009.0.14
env = /apps/usr/env/cfdace-2009.0.14.sh
Intel compilers 11.0.083
path = /apps/pkg/intel-11.0.083
env = /apps/usr/env/intel-11.0.sh
*Matlab R2009a
path = /apps/pkg/apps/pkg/matlab-R2009a
env = (n/a; will be in default path)
PGI compilers 9.0-1
path = /apps/pkg/pgi/linux86-64/9.0-1
env = /apps/usr/env/pgi-9.0.sh
SAS 9.2
path = /apps/pkg/sas-9.2
*STARCCM+ 4.02
path = /apps/pkg/sc+4.02
env = /apps/usr/env/starccm+4.02.007.sh

* there are example torque scripts for these apps in /apps/torque/examples

Application versions to be retired:

ABAQUS 6.5-1
ABAQUS 6.6-1
CFDACE 2009
Intel compilers 9.1.047
Matlab R2007b
PGI compilers 7.04
STARCCM+ 3.06

The semester begins on August 17th; our plan is to retire the old application versions on August 15th. Please make sure that any scripts that reference the old versions are updated with the new versions. The new applications are on the system and available for use now, so you can start using them as soon as you like. I provide the path and environment file for the new apps as a reference for updating your scripts if need be.

If you have any questions or concerns regarding any of the applications mentioned above, please let us know by emailing URCSupport@uncc.edu.

We are beginning to transition from the Condor Job Scheduler that we have used in the cluster since its inception in 2005 to the Torque/Maui Scheduling system which has become the predominate scheduler for academic HPC environments.  This change will allow us to take better advantage of tools and techniques developed at other universities, provide better integration with commercial products which support Torque/Maui for parallel processing (like MatLab), and improve our queue management by implementing resource limits or “fair share” algorithms.

This change to the scheduler will require all users to replace their condor submit scripts with similar torque submit scripts and switch from using the condor commands (condor_submit, condor_q, etc.) to manage cluster jobs to using the equivalent torque commands (qsub, qstat, etc.)   This change is conceptually very simple, but does require learning a few new techniques. The URC staff has prepared a web page with instructions for getting started with torque, as well as, sample torque scripts for many of our most common applications (see folder /apps/torque/examples on the cluster). The staff is also available to assist users with the transition as needed – simply send an email request to URCSupport@uncc.edu.

In order to allow this transition to occur with the minimum possible disruption, we will run both schedulers for several months and slowly move the compute nodes from the condor pool into the torque pool as our users make the necessary adjustments.  Currently, we have 32 computing cores available in the URC torque pool and 120 computing cores in the corresponding condor pool with additional nodes available in pools that are dedicated to specific research groups or activities (e.g. the Mechanical Engineering nodes).  You can see a summary of the activity in the condor and torque pools on the URC home page.  Once all users and compute nodes have been moved over to torque, we will discontinue our use of condor.

Email questions or comments to URCSupport@uncc.edu .

As part of URC’s efforts to provide users with a more user-friendly and efficient environment, we are in the process of transitioning our job scheduler from Condor to Torque/Maui.  Torque is an Open Source scheduler based on the old PBS scheduler code.

The following is a set of directions to assist a user in learning to use Torque to submit jobs to the URC cluster(s).  It is tailored specifically to the URC environment and is by no means comprehensive.  Details not found in here can be found online at:

http://www.clusterresources.com/resources/documentation.php/

as well as man pages for the various command.

Note:
Some of the sample scripts displayed in the text are not complete so that the reader can focus specifically on the item being discussed.  Full, working examples of scripts and commands are provided in the Examples section at the end of this document.

Configuration

Before submitting jobs, it is important to understand how the compute clusters are laid out in terms of Torque scheduling.

Like the Condor configuration, it is replacing, Torque at URC will accept jobs submitted from three hosts:

submit.urc.uncc.edu    (General URC users)
mees.urc.uncc.edu    (MEES users)
mees10.urc.uncc.edu    (MEES users – Dr Uddin)

As with Condor, users will use SSH to connect to one of the hosts above and from there submit the various Torque specific commands as outlined below. Also, submission of jobs to Torque will also be supported by the URC portal.

All jobs submitted from the submit host are funneled to the URC Torque server (m03) which is running the Torque and Maui server processes.

Compute nodes in the cluster(s) are logically grouped and accessed via Torque “queues.”  Users and groups of users are granted the rights to submit jobs to specific queues and hence run on specific nodes.  The currently defined queues are:

batch    -    Default queue.  (Disabled)
urc    -    queue for the general URC users.
mees    -    queue for the MEES users.
mees10-    queue for MEES users – Dr. Uddin)
wrf    -    queue for the WRF project group

To determine if a specific user has been granted rights to submit to a particular queue, use the Torque command qmgr:

$ qmgr -c "list queue QNAME acl_users" m03

where QNAME is one of the queues defined above.  Note that all users have access to submit to the “urc” queue.  The access control list is NOT enabled for that queue.

Submitting a Job

Scheduling a job in Torque is similar to the method used in URC’s previous scheduler (Condor).  It requires creating a file that describes the job (in this case a shell script) and then that file is given as an argument to the Torque command “qsub” to execute the job.

First of all, here is a sample shell script (myjob.sh) describing a simple job to be submitted:

#! /bin/bash

# ==== Main ======
/bin/date

This script simply runs the ‘date’ command.  To submit it to the scheduler for execution, we use the Torque qsub command:

$ qsub  myjob.sh

This will cause the script (and hence the date command) to be executed on the default queue.

The simple example above can be expanded to demonstrate additional options:

$ qsub -N "MyJob"  -q "urc"  my_script.sh

In this example, the “-n” switch gives the job a name while the “-q” switch is used to route the job to the “urc” queue.

Many of the command line options to qsub can also be specified in the shell script itself using Torque (PBS) directives. Using the previous example, our script (my_script.sh) could look like the following:

#!/bin/sh

# ===== PBS OPTIONS =====
### Set the job name
#PBS -N "MyJob"

### Run in the queue named "urc"
#PBS -q "urc"

# ==== Main ======
/bin/date

Running the command is now simply:

$ qsub my_script.sh

For the entire list of options, see the man page qsub i.e.

$ man qsub

Standard Output and Standard Error
In  Torque, any output that would normally print to stdout or stderr is collected into two files. By default these files are placed in the initial working directory where you submitted the job from and are named:

scriptname.{o}jobid for stdout
scriptname.{e}jobid for stderr

In our previous example (if we did not specify a job name with -n) that would translate to:

My_script.sh.oNNN
My_script.sh.oNNN

Where NNN is the job ID number returned by qsub.  If I named the job with -n (as above) and it was assigned job id 801, the files would be:

MyJob.o801
MyJob.e801

Note:
Torque accomplishes this by buffering the output on the execution host until the job completes and then copies it back to the working directory.   An unfortunate circumstance of this is that if your job does write information to stdout or stderr, you would be unable to view it until the job completes.

The qsub command does have an option (-k) to “keep” the files from being buffered and spooled in this manner.  Unfortunately, it will only allow the files to be created in the user’s home directory ($HOME) rather than in the working directory.  This is not very convenient in most cases.

To avoid this type of problem, URC suggests that (rather than using the -k option) that in your shell script (My_script.sh), you should explicity direct stdout and stderr to files.  There are many ways to do this in a shell script.  Some common ways are:

#! /bin/bash
exec 1>$PBS_O_WORKDIR/out  2>$PBS_O_WORKDIR/err

which will direct stdout and stderr to the current working directory of the qsub command.

Another method is to enclose curly braces {} around the body of text in the script and recirect it:

#! /bin/bash

{
/bin/date
} > $PBS_O_WORKDIR/out 2>$PBS_O_WORKDIR/err

See the examples below for a more detailed example.

Monitoring a Job

Monitoring a Torque job is done primarily using the Torque command “qstat.” For instance, to see a list of available queues:

$ qstat -q

To see the status of a specific queue:

$ qstat "queuename"

To see the full status of a specific job:

$ qstat -f  jobid

where jobid is the unique identifier for the job returned by the qsub command.

Deleting a Job

To delete a Torque job after it has been submitted,  use the qdel command:

$ qdel jobid

where jobid is the unique identifier for the job returned by the qsub command.

Monitoring Compute Nodes

To see the status of the nodes associated with a specific queue, use the torque command pbs_nodes(1) (qlso refered to as qnodes):

$ pbsnodes :queue_name

where  queue_name is the name of the queue  prefixed by a colon (:).  For example:

$ pbsnodes :urc

would display information about all of the nodes associated with the “urc” queue.  The output includes (for each node) the number of cores available (np= ).  If there are jobs running on the node, each one is listed in the (jobs= ) field.  This shows how many of the available cores are actually in use.

Parallel (MPI) Jobs

Parallel jobs are submitted to Torque in the manner described above except that you must first ask Torque to reserve the number of  processors (cores) you are requesting in your job.  This is accomplished using the -l switch to the qsub command:

For example:

$ qsub  -q urc -l nodes=16 my_script.sh

would submit my script requesting 16 processors (cores)  from the “urc” queue.  The script (my_script.sh) would look something like the following:

#! /bin/bash
mpirun -hostfile $PBS_NODEFILE  my_mpi_prgram

If you need to specify a specify number of processors (cores) per compute host, you can append a colon (:) to the number of specified nodes and then append the number of processors per host.  For example, to request 16 total processors (cores) with only 4 per compute host, the syntax would be:

$ qsub  -q urc -l nodes=4:ppn=4 my_script.sh

In this example, mpirun is using the environment variable $PBS_NODEFILE as the path to the hosts file that contains the list of nodes that MPI job can run on.  This variable is automatically set by Torque and the file it points to is automatically populated as a result of the “-l nodes=16″ argument given to the qsub command.

Note that the syntax of the mpirun command varies across MPI implementations.  The $PBS_NODEFILE is set by Torque and so can be used by any implementation of mpirun provided that the proper syntax is used.  The example above (–hostfile) is for OpenMPI.

As described previously, options to qsub can be  specified directly in the script file.  For the example above, my_script.sh would look similar to the following:

#! /bin/bash
# ===== PBS OPTIONS =====
### Set the job name
#PBS -N MyJob

### Run in the queue named "urc"
#PBS -q urc
### Specify the number of cpus for your job.
#PBS -l nodes=4:ppn=4

mpirun -hostfile $PBS_NODEFILE  my_mpi_prgram

Examples of Torque Submit Scripts

NOTE: Additional sample scripts can be found online in /apps/torque/examples.

[1] Simple Job

#! /bin/bash

# ===== PBS OPTIONS =====
### Set the job name
#PBS -N MyJob
### Run in the queue named "urc"
#PBS -q urc

# ===== END PBS OPTIONS =====

# Redirect standard out and standard error.
exec 1>$PBS_O_WORKDIR/$PBS_JOBID.out 2>$PBS_O_WORKDIR/$PBS_JOBID.err

# Main Program
/bin/date

[2] Parallel Job – 16 Processors (Using OpenMPI)

#! /bin/bash

# ===== PBS OPTIONS =====
### Set the job name
#PBS -N MyJob
### Run in the queue named "urc"
#PBS -q urc
### Specify the number of cpus for your job.
#PBS -l nodes=16
# ===== END PBS OPTIONS =====

# Redirect standard out and standard error.
exec 1>$PBS_O_WORKDIR/$PBS_JOBID.out 2>$PBS_O_WORKDIR/$PBS_JOBID.err

# =========== Main Program ===========
# Setup to use OpenMPI
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/apps/sys/openmpi-1.2.4/rhel5_u1-64/gnu/lib
MPI_RUN=/apps/sys/openmpi-1.2.4/rhel5_u1-64/gnu/bin/mpirun

# Run the program "simplempi" with an argument of "30"
$MPI_RUN --hostfile $PBS_NODEFILE /users/joe/simplempi 30

ITS is planning a network outage for Saturday March 14th from 6am until 9am.  As a result you will be unable to reach the ChargerNet computing cluster during this time.  The good news is that we expect the cluster to remain operational so your running jobs should not be affected.  The one exception is jobs that use the Abaqus software package – they may fail during the outage because they will be unable to contact the license server which is provided by the College of Engineering.

So, if you are using Abaqus, you may want to plan your work so that your jobs finish prior to March 14th.  If not, then the only impact you will see is loss of access to the cluster for 3 hours on the morning of March 14th.

We apologize for any inconvenience that this may cause, but the need to perform certain routine maintenance tasks requires a complete outage once or twice per year.  We do not anticipate the need to schedule any additional outages until spring break.

Please send a response if you have any questions or comments.