[English|Japanese]

Wide-Area Coordinated Cluster System for Education and Research

Operation ended on August 2019. Please refer to the top page for the new system.

Test operation commenced on August 1, 2014.

Logging In

Log in to the development server using an SSH client such as TeraTerm or putty. The different hostnames of the server are prepared for internal users and external users.

The development server is a group of servers to develop and compile programs. Do not perform large-scale calculations on the server. Conduct the actual calculation on the cluster as a job via Torque.

Note that the development server is redundantly configured using two servers to distribute the loading.

For Internal Users

The hostname of the development server is wdev.edu.tut.ac.jp. Enter your account information provided by the Information and Media Center in the user name and password field.

$ ssh wdev.edu.tut.ac.jp

For External Users

The hostname of the development server is lark.imc.tut.ac.jp. Use an account provided through the account registration system (par + 7-digit numerals) to log in to the server. External users are identified through a public key authentication method. Register your public key on the Change Profile page. Refer to the How to use SSH client page for the procedure for creating a public key.

$ ssh lark.imc.tut.ac.jp

Queue Configuration

The following is the tentative configuration.

System Configuration

Hardware Configuration

* It is not possible to execute jobs using wsnd17 and wsnd31.

File System Configuration

The work area can also be viewed through /work/user_name/.

Compiler

Message Passing Interface (MPI)

Software Configuration

・You must be a Type A user to use ANSYS, ABAQUS, Patran, DEFORM-3D, COMSOL, GAUSSIAN, CHEMKIN-PRO, and MATLAB.

・To apply for registration as a Type A user, see: http://imc.tut.ac.jp/en/research/form.

Using the Software

For using the software, see Using Cluster Systems.

Using the Xeon Phi Processor

Job execution using the Xeon Phi processor can be performed in native mode or offload mode. In native mode, the Xeon Phi processor is used as a single calculation node. The MPI program can be used as it is. In offload mode, a specific process within the source code can be offloaded to the Xeon Phi processor and executed. The Xeon Phi processor can be used in the same manner as GPGPU through OpenACC.

Native Execution

Sample Source Program

sample_phi.c

#include <stdio.h>
#include <mpi.h>
int main(int argc, char **argv)
{
    int myid, nprocs;
    char hname[128]="";

    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
    MPI_Comm_rank(MPI_COMM_WORLD,&myid);

    gethostname(hname,sizeof(hname));
    if (myid==0)
        printf ("NUMBER OF PROCESSES: %3d\n", nprocs);
    printf("HELLO WORLD! (HOST NAME: %10s, MYID: %3d)\n", hname, myid);

    MPI_Finalize();

    return 0;
}

* The source content is the same as the sample source program for MPI (sample_c_mpi.c). Only the file name has been changed.

Compiling

It is necessary to create an execution file for the Xeon CPU and another execution file for the Xeon Phi coprocessor. Use the Intel compiler.

Creating the execution file for the Xeon CPU

% mpiicc sample_phi.c -o sample_phi

Creating the execution file for the Xeon Phi coprocessor

% mpiicc -mmic sample_phi.c -o sample_phi.mic

Notes

Add “.mic” to the execution file for the Xeon Phi coprocessor.

Apart from “.mic”, the Names of the execution files for the Xeon CPU and the Xeon Phi coprocessor must be identical.

Sample Script to Submit Jobs

phi_native.sh

### sample

#!/bin/sh
#PBS -q wSrchq
#PBS -l nodes=3:ppn=2:Phi

MIC0_PROCS=3
MIC1_PROCS=1
source /common/torque/MIC/mkmachinefile.sh

cd $PBS_O_WORKDIR
mpirun -machinefile ${MACHINEFILE} -n ${PBS_NP} ./sample_phi

Note on #PBS -q

A job can be submitted to either the wSrchq or wLrchq queues.

Note on #PBS -l

Always specify :Phi in #PBS –l. This is the setting to specify the calculation node equipped with the Xeon Phi processors.

Note on MIC0_PROCS and MIC1_PROCS

Two Xeon Phi processors are installed on a single node in the system. MIC0_PROCS= and MIC1_PROCS= are used to specify the number of processes to invoke on the respective Xeon Phi processor on the calculation node. In the script file shown above, three calculation nodes are used by specifying nodes=3. One of the Xeon Phi processors on each node invokes 3 processes while the other invokes 1 process. The Xeon Phi’s capacity is 60 core/240 threads per processor. The value specified in MIC0_PROCS= and MIC1_PROCS= must be 240 or less.

Note on the number of parallel process

In the script file shown above invokes 18 processes in total and executes them in parallel. Assume that wsnd00, wsnd02, and wsnd03 are selected by nodes=3. To distinguish the two Xeon Phi processors on the calculation node, they are named Xeon Phi0 and Xeon Phi1, respectively. A total of 18 processes are invoked in this example: 2 processes in wsnd00, 3 processes in Xeon Phi0 in wsnd00, 1 process in Xeon Phi1 in wsnd00, 2 processes in wsnd02, 3 processes in Xeon Phi0 in wsnd02, 1 process in Xeon Phi1 in wsnd02, 2 processes in wsnd03, 3 processes in Xeon Phi0 in wsnd03, and 1 process in Xeon Phi1 in wsnd03.

Others

Change the values for the following options as appropriate: the queues to submit the job, the values for nodes=，ppn=, MIC0_PROCS=, and MIC0_PROCS=, and the execution file name (sample_phi in the above script file).

Offload Execution

Sample Source Program

tbo_sort.c

Location: /common/intel-2013SP1/composer_xe_2013_sp1.0.080/Samples/en_US/C++/mic_samples/LEO_tutorial/tbo_sort.c

Compiling

When compiling, always specify the -openmp option. Specify an Intel complier.

% icc -openmp

tbo_sort.c -o tbo_sort

Sample Script to Submit Jobs

phi_offload.sh

### sample

#!/bin/sh
#PBS -q wSrchq
#PBS -l nodes=1:ppn=20:Phi

cd $PBS_O_WORKDIR
./tbo_sort

Note on #PBS -q

A job can be submitted to either the wSrchq or wLrchq queues.

Note on #PBS -l

There is no need to change #PBS -l nodes=1:ppn=20:Phi. This line means to use a single calculation node on Xeon Phi exclusively.

Others

Change the queue to submit the job and the execution file name (tbo_sort in the above script file) as appropriate.

Limitations

● Specify the IP address of Xeon Phi when using mpirun –machinefile. Usually, the value for –machinefile is generated automatically by mkmachinefile.sh.

● Only tcp is available as the MPI communication. tcp is used by default in mkmachinefile.sh.

● Native execution is available on wsnd 00 to 09, 14, and 15 as of October 2015. The job scheduler automatically chooses the node unless otherwise specified explicitly.