In general, the user account name is the same as the official campus login ID (also used for Novell, 49er Express, and Mosaic); however, the account password is unique to the URC cluster and is not synced with any other campus authentication system.

In addition to kpasswd, the user can also change their password when logged into the ChargerNet Portal by clicking on the “Change Password” link in the Utilities menu.

The supported methods for connecting to this node are:

• Secure Shell (SSH) - For secure remote login
• Secure Copy (SCP) or Secure FTP (SFTP) – For secure data transfer.

Most common Unix or Linux operating systems (including Mac OS X) come equipped with client versions of these programs for use from the command line. For more information refer to the man pages: ssh, scp, sftp (these are for the Redhat Linux clients – other Unix or Linux systems should be similar but may not be identical).

There are also various commercial and shareware versions of these commands available for MS Windows. One popular free client is available from ssh.com

Parallel processing is supported via two separate Condor universes. The older MPI universe supports the mpich implementation of the Message Passing Interface (MPI) standard while the newer Parallel universe provides a flexible mechanism that can be adapted to almost any parallel execution methodology.

Parallel Execution in the MPI Universe

If you wish to use the MPI universe to run your own mpi code, you need to include the mpich libraries (version 1.2.4 only) when compiling/linking your program. One way to do this is to use the mpich compiler commands mpicc, mpiCC, mpif77, or mpif90 provided in /apps/mpich/bin. These commands are simply scripts that run the gnu C or C++ compiler or the Portland Group Fortran compiler with the recommended libraries and command line options. Alternatively, you can simply reference the relevant include files and libraries (found in /apps/mpich/include and /apps/mpich/lib respectively) in your project makefile or on the command line.

Many open source or commercial applications also include support for parallel processing. In order for the applications to work properly in the MPI universe, they must support mpich version 1.2.4. Applications that do not support mpich 1.2.4 should be run via the parallel universe.

Once you have a properly compiled mpich executable, the process to submit a job to Condor is as described above

condor_submit -n mpi <submit_file>

where the submit file would be as shown above with the Universe set to MPI and an additional line to specify the desired number of processors, e.g.:

Machine_count = 4

Choosing an optimal value for the number of processors often requires some experimentation since each application is likely to scale differently depending on a variety of factors like the ratio of computation to I/O. Very few applications scale well beyond 16 or 32 processors.

The additional qualifier on the condor_submit command (-n mpi) is necessary to direct the job to the correct condor scheduling host. Not all universes are managed by the same condor server.

Parellel Execution in the Parallel Universe

Parallel execution in the “parallel” universe is extremely flexible in terms of the types of parallel operation that are possible. The user specifies a wrapper script in addition to an executable. The wrapper is executed on each node that is assigned to the user’s job and is passed the executable as an argument. This allows the user to construct a custom parallel environment within their assigned nodes, and then start the real executable in a fashion that is appropriate to their custom environment.

For users of the old mpi queue, the changes required to switch to the parallel queue are minimal. In the condor_submit file change these lines:

universe = mpi
executable = my.exe
arguments = -arg1 -arg2

with these lines:

universe = parallel
executable = /apps/condor/scripts/mpi-1.2.4
arguments = my.exe -arg1 -arg2

The universe changes from mpi to parallel, the user’s executable becomes the first argument, and the executable is a standard wrapper script that is installed as part of the condor software package.

The resulting submit file is then submitted using:

condor_submit -n parallel <submit file name>

In this example the user’s job will run using version 1.2.4 of mpich just as it does in the mpi queue. For users who wish to use a later version of mpich, a separate wrapper script, mpi-1.2.5, is also available under /apps/condor/scripts. Additional, wrapper scripts will be added whenever new versions of mpich are installed.

Additional wrapper scripts have been developed to support ABAQUS and Matlab DCE, and will be developed as need to support other applications (e.g. LAM MPI).

We are putting the finishing touches on our latest cluster: “VIPER”. We are excited about getting users up-and-running on it, which should be within the next couple of weeks. Here is some information about VIPER, so that you can familiarize yourself with this new computing resource.

VIPER consists of

• 60 compute nodes
• 480 computing cores
• 5 TFlops sustained
• 58 TBs storage
• 1.4 TBs RAM

VIPER will initially be split into 2 distinct queues:

• “VIPER Job Queue”
• Parallel and single-threaded jobs
• 54 compute nodes
• 432 computing cores
• 4.5 TFlops sustained
• Dual 2.93 GHz quad-core Intel Xeon X5570 Processors
• 24 GBs RAM (3 GBs/core)
• 143 GB hard drive for local scratch storage
• 20 Gb/s Infiniband HCA
• 1Gb/s Ethernet NIC

• “Single-threaded Job Queue”
• Single-threaded jobs exclusively
• 6 compute nodes
• 48 computing cores
• 500 GFlops sustained
• Dual 3.0 GHz quad core Intel Xeon E5450 Processors
• 16 GBs RAM (2GBs/core)
• 1 Gb/s Ethernet NIC

VIPER Storage will consist of three performance tiers:

• 58 TBs of usable network attached storage
• replicated, RAID 6 storage for data retention
• non-replicated, RAID6 for run-time data storage
• a Lustre parallel file system cluster for high-performance IO.

The VIPER Queue will have an Infiniband cluster interconnect for increased performance in Parallel jobs:

• 3.8 Tb/s total bandwidth
• 1 microsecond latency

Changes and Improvements in VIPER

As you can see from the above description, there are a lot of hardware differences between VIPER and the original URC cluster. In addition to the hardware differences, there are several other key differences that we would like to outline below in order to make your transition as smooth as possible.

• Campus Authentication.
  We now allow you to use your campus (i.e., NinerNet) password to log in to VIPER. Now you have one less password to keep up with! (For a very limited transition period, existing cluster users may use their NinerNet password or their URC Cluster password in order to log in).
• New storage structure and quotas.
  All users will have a 50GB home directory quota (in /users/username). All home directories will be backed up nightly during the week as well as backed up weekly. Users will also have a “compute” storage area of 100GB that is NOT BACKED UP (in /projects/username). Users are free use this area as needed, but we encourage you to save off all important output to your home directory if you wish to have it backed up. We also offer some high-speed scratch storage (/scratch) that has no quota and is free to use by all users. This area is meant for temporary storage of data required by current jobs, and is not intended for long-term storage. Please do not store anything in the scratch area that is important or irreplaceable. The Research Computing Group reserves the right to remove data from the scratch area, on an “as-needed” basis, to prevent the volume from filling up.
• Environment Modules
  In order to help manage user environments, we have implemented “Environment Modules”. This is a very popular method used in HPC environments for customizing a user’s environment specific to the applications that he or she uses. You can find more information on our Environment Modules help page.
• Increased Cores/User
  Users will be able to use up to 96 cores at a time. So, if you require 96 cores for a single parallel job, You can run it on VIPER. If your jobs require only a single core each, you will be able to run up to 96 jobs at a time. (so long as cores are available!)
• Separate Submit and Interactive Hosts
  On the original URC cluster, the submit host (submit.urc.uncc.edu) was used not only to submit jobs, but it was also used for interactive application usage. There have been times when a user’s application would use up most of the resources on the submit server, making it difficult (or impossible) for others to get logged in and submit jobs. For the VIPER cluster, we decided to have 2 hosts that you can log in to: a submit server and an interactive server. The submit server, “viper.urc.uncc.edu,” should only be used for submitting jobs to the cluster, while the interactive server, “iviper.urc.uncc.edu,” is similarly configured as one of the compute nodes and it meant for users to run applications interactively on.

The environment modules are a system that allows you to quickly and easily modify your shell environment to access different software packages. Research Computing offers a large (and growing) number of software packages to users, and each package may contain several tools, manual pages and libraries, or it may require special setup to work properly. Some software packages come in several versions or flavors, many of which conflict with each other. Modules allows you to tailor your shell to access exactly the packages you need by setting up the relevant environment variables for you, and automatically avoiding many possible conflicts between packages.

Command Summary

module avail

module load 

module unload 

module whatis 

module list

module purge

module display 

Example Usage

$ module avail

---------------------------- /apps/usr/modules/mpi -----------------------------
openmpi/1.3.3(default) openmpi/1.3.3-intel    openmpi/1.3.3-pgi

------------------------- /apps/usr/modules/compilers --------------------------
intel/10.1          intel/11.0-32bit    pgi/9.0(default)
intel/10.1-32bit    pgi/7.1             pgi/9.0-32bit
intel/11.0(default) pgi/7.1-32bit

---------------------------- /apps/usr/modules/apps ----------------------------
cfdace/2009.0.14       netcdf/3.6.2-intel     netcdf/4.0.1-pgi
cfdace/2009.2(default) netcdf/3.6.2-pgi       starccm/4.02(default)
epanet2/2.0.12         netcdf/4.0.1(default)
netcdf/3.6.2           netcdf/4.0.1-intel

--------------------- /apps/sys/Modules/3.2.6/modulefiles ----------------------
dot         module-cvs  module-info modules     null        use.own

$ module load intel/11.0

$ module list
Currently Loaded Modulefiles:
  1) modules      2) intel/11.0

How the Modules are Organized and Grouped

The modules are organized into “categories”, which include: /apps/usr/modules/mpi, compilers, apps, and /apps/sys/Modules/3.2.6/modulefiles. Under each category, you will see “groups” of applications: openmpi, intel, pgi, to name a few. Within each group, there may be several versions to choose from. The group and version are separated with a “slash” (/).

Default Modules

You probably noticed some modules listed above are suffixed with a “(default)”. The “default” module is the module that will get loaded if you do not specify a version number. For example, we can load the “intel/11.0″ module by omitting the version number:

$ module load intel

$ module list
Currently Loaded Modulefiles:
  1) modules      2) intel/11.0

Note: If you plan to load a version of a module that is not the default, then you must specify the version in the module load command.

Conflicts and Prerequisites

Some modules conflict with others, and some modules are prerequisites of others. Modules handles both scenarios.

The following is an example of trying to load a module that is dependent upon another:

$ module load openmpi/1.3.3-pgi

openmpi/1.3.3-pgi(12):ERROR:151: Module 'openmpi/1.3.3-pgi' depends on one of the module(s) 'pgi/9.0'
openmpi/1.3.3-pgi(12):ERROR:102: Tcl command execution failed: prereq pgi/9.0

To resolve the above error, simply load the “prereq” module first, then load the original module. You can do this in a single command:

$ module load pgi/9.0 openmpi/1.3.3-pgi

More information

You can find more information about Environment Modules on SourceForge.net:
http://modules.sourceforge.net/

Or from the man pages (under “Misc” menu):
http://www.urc.uncc.edu/urc/man-pages/

Access to the ChargerNet File System is available for those users who need to access their home directory or project area from a remote client (on campus or via the campus VPN).

URC maintains a Samba server which allows SMB capable clients (i.e. MS-Windows, MacOS X, and Linux) to mount remote shares as a local drive or mount point.  To obtain permission to use this server, send email to URCSupport@uncc.edu.

Below are instructions for mounting a ChargerNet Samba Share from  various client machine types. Note that the Samba server  authenticates using Campus (NinerNet) passwords.

The examples below assume the following for ChargerNet user “joeuser”:

ChargerNet Samba Server:                        samba.urc.uncc.edu

ChargerNet login:                                         joeuser   (its.uncc.edu/joeuser)*

ChargerNet home directory:                      /users/joeuser

ChargerNet share name:                            joeuser

[ MS-Windows XP ]

1. Select ‘My Computer’

2. Select ‘Tools – > Map Network Drive’

3. Fill in path to folder (share)  i.e.  \\samba.urc.uncc.edu\joeuser

4. Select ‘Connect using a different user name’

5. Fill in username* (see footnote below) i.e    its.uncc.edu\joeuser

6. Fill in NinerNet password.

[ MacOS X ]

1. In ‘Finder’ application select Go -> Connect to Sever

2. Fill in Server address (and share) indicating smb protocol i.e. smb://samba.urc.uncc.edu/joeuser

3. Press ‘Connect’

4. Fill in Name (ChargerNet login)  i.e. joeuser (domain not required)

5. Fill in NinerNet password

[ Red Hat Linux - File Browser ]

1. In ‘File Broswer‘ application select File -> Connect to Server

2. Fill in the following:

Service Type: Windows share

Server: samba.urc.uncc.edu

Share: joeuser

Domain Name: its.uncc.edu

Name to use for connection: joeuser

3. Press ‘Connect’ button

4, Fill in NinerNet password.

[ Red hat Linux - Command Line ]

You can use the mount.cifs(8) command which is part of the samba(7) suite.  See the man page for mount.cifs for more details

To mount a share (to /home/joeuser/samba on the client) use the following command:

$ /sbin/mount.cifs //samba.urc.uncc.edu/joeuser  /home/joeuser/samba -o user=joeuser

To unmount the share:

$/sbin/umount.cifs /home/joeuser/samba

* Objects on the network including servers and even users are distinguished using “domain” naming conventions.  For instance, the samba server referenced above belongs to URC and is therefore part of the ‘urc’ domain.  Hence the full name of that object is: samba.urc.uncc.edu.

However, since the samba server uses the campus NinetNet system to authenticate user logins, the full user name (where required as in the case of a MS-Windows client)  uses the ‘its’ domain not the ‘urc’ domain.  Hence, a fully qualified user name is specified as:  its.uncc.edu\joeuser.