Linux High performance Computing Implementation Guide. By Syed Asim Abbas with help MR Kamarn Azeem CBTs.

For comments and suggestions : asimabbas31@gmail.com

HPCC Installation

I have 4 node for HPCC. 2 nodes are HP ML 360 servers with 2 x dual core processor. One is 1.7 mhz processor and other is AMD Sempron(tm) Processor 2600+

1) Centos 5.2 Installation on all nodes

2) Stop unnecessary services

3) /etc/hosts

vi /etc/hosts (hpcc2)

Do not remove the following line, or various programs

that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

192.168.1.180 hpcc1.isb.ots.com hpcc1

192.168.1.49 hpcc2.isb.ots.com hpcc2

192.168.1.69 test.isb.com test

192.168.1.40 nworkhorse.isb.ots.com nworkhorse

vi /etc/hosts (test)

Do not remove the following line, or various programs

that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

192.168.1.180 hpcc1.isb.ots.com hpcc1

192.168.1.49 hpcc2.isb.ots.com hpcc2

192.168.1.69 test.isb.com test

192.168.1.40 nworkhorse.isb.ots.com nworkhorse

vi /etc/hosts (nworkhorse)

Do not remove the following line, or various programs

that require network functionality will fail.

127.0.0.1 localhost.localdomain localhost

192.168.1.180 hpcc1.isb.ots.com hpcc1

192.168.1.49 hpcc2.isb.ots.com hpcc2

192.168.1.69 test.isb.com test

192.168.1.40 nworkhorse.isb.ots.com nworkhorse

root@hpcc1 ~]# ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/root/.ssh/id_dsa):

/root/.ssh/id_dsa already exists.

Overwrite (y/n)?

[root@hpcc1 ~]# ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/root/.ssh/id_dsa):

/root/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_dsa.

Your public key has been saved in /root/.ssh/id_dsa.pub.

The key fingerprint is: [root@hpcc2 ~]# ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/root/.ssh/id_dsa):

/root/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_dsa.

Your public key has been saved in /root/.ssh/id_dsa.pub.

The key fingerprint is:

3a:c0:dc:ad:aa:52:07:f8:43:b2:cb:ee:81:c8:6a:8f root@hpcc2.isb.ots.com

[root@hpcc2 ~]# scp ~/.ssh/id_dsa.pub root@hpcc1:/root/.ssh/authorized_keys

root@hpcc1’s password:

id_dsa.pub 100% 612 0.6KB/s 00:00

[root@hpcc2 ~]#

[root@hpcc2 ~]#

[root@hpcc2 ~]#

[root@hpcc2 ~]#

[root@hpcc2 ~]# ssh hpcc1

Last login: Tue May 5 11:02:26 2009 from 192.168.1.19

[root@hpcc1 ~]# logout

7b:a6:db:3e:55:ce:da:6b:ce:48:28:d3:9a:29:84:82 root@hpcc1.isb.ots.com

[root@hpcc1 ~]# scp ~/.ssh/id_dsa.pub root@hpcc2:/root/.ssh/authorized_keys

root@hpcc2’s password:

id_dsa.pub 100% 612 0.6KB/s 00:00

[root@hpcc1 ~]# ssh hpcc2

Last login: Wed May 13 11:03:40 2009 from hpcc1.isb.ots.com

[root@hpcc2 ~]# logout

Connection to hpcc2 closed.

NTP: for all nodes :


Node 1

[root@hpcc2 ~]# vi /etc/ntp.conf

[root@hpcc2 ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [FAILED]

Starting ntpd: [ OK ]

[root@hpcc2 ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

127.127.1.0 .LOCL. 10 l 5 64 1 0.000 0.000 0.001

[root@hpcc2 ~]#

Node 2

[root@hpcc2 ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@hpcc2 ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

127.127.1.0 .LOCL. 10 l 1 64 1 0.000 0.000 0.001

[root@hpcc2 ~]# vi /etc/ntp.conf

[root@hpcc2 ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@hpcc2 ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

10.0.0.1 LOCAL(0) 11 u - 64 1 0.155 -691264 0.001

127.127.1.0 .LOCL. 10 l - 64 0 0.000 0.000 0.001

[root@hpcc2 ~]#

Node 3

[root@test ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@test ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

127.127.1.0 .LOCL. 10 l 1 64 1 0.000 0.000 0.001

[root@test ~]# vi /etc/ntp.conf

[root@test ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@test ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

192.168.1.180 LOCAL(0) 11 u - 64 1 0.155 -691264 0.001

127.127.1.0 .LOCL. 10 l - 64 0 0.000 0.000 0.001

[root@test ~]#

[root@nworkhorse ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@nworkhorse ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

127.127.1.0 .LOCL. 10 l 1 64 1 0.000 0.000 0.001

[root@nworkhorse ~]# vi /etc/ntp.conf

[root@nworkhorse ~]# /etc/init.d/ntpd restart

Shutting down ntpd: [ OK ]

Starting ntpd: [ OK ]

[root@nworkhorse ~]# ntpq -p -n

remote refid st t when poll reach delay offset jitter

==============================================================================

192.168.1.180 LOCAL(0) 11 u - 64 1 0.155 -691264 0.001

127.127.1.0 .LOCL. 10 l - 64 0 0.000 0.000 0.001

[root@nworkhorse ~]#

PDSH:


[root@hpcc1 ~]# wget http://nchc.dl.sourceforge.net/sourceforge/pdsh/pdsh-2.18-1.src.rpm

–11:52:48– http://nchc.dl.sourceforge.net/sourceforge/pdsh/pdsh-2.18-1.src.rpm

Resolving nchc.dl.sourceforge.net… 211.79.60.17, 2001:e10:ffff:1f02::17

Connecting to nchc.dl.sourceforge.net 211.79.60.17 :80… connected.

HTTP request sent, awaiting response… 200 OK

Length: 462342 (452K) [application/x-wais-source]

Saving to: `pdsh-2.18-1.src.rpm’

100%[======================================================================================================>] 462,342 6.43K/s in 16s

11:53:05 (28.9 KB/s) - `pdsh-2.18-1.src.rpm’ saved [462342/462342]

[root@hpcc1 ~]# scp pdsh-2.18-1.src.rpm hpcc2:/root

pdsh-2.18-1.src.rpm 100% 452KB 451.5KB/s 00:00

[root@hpcc1 ~]#

[root@hpcc1 ~]# rpmbuild –rebuild pdsh-2.18-1.src.rpm

Installing pdsh-2.18-1.src.rpm

warning: user grondo does not exist - using root

warning: group grondo does not exist - using root

warning: user grondo does not exist - using root

warning: group grondo does not exist - using root

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.4710

  • umask 022

  • cd /usr/src/redhat/BUILD

  • LANG=C

[root@hpcc2 ~]# rpmbuild –rebuild pdsh-2.18-1.src.rpm

Installing pdsh-2.18-1.src.rpm

warning: user grondo does not exist - using root

warning: group grondo does not exist - using root

warning: user grondo does not exist - using root

warning: group grondo does not exist - using root

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.4710

  • umask 022

  • cd /usr/src/redhat/BUILD

  • LANG=C

[root@hpcc1 i386]# rpm -ivh pdsh-*

Preparing… ########################################### [100%]

1:pdsh-rcmd-ssh ########################################### [ 14%]

2:pdsh ########################################### [ 29%]

3:pdsh-debuginfo ########################################### [ 43%]

4:pdsh-mod-dshgroup ########################################### [ 57%]

5:pdsh-mod-machines ########################################### [ 71%]

6:pdsh-mod-netgroup ########################################### [ 86%]

7:pdsh-rcmd-exec ########################################### [100%]

[root@hpcc2 i386]# rpm -ivh pdsh-*

Preparing… ########################################### [100%]

1:pdsh-rcmd-ssh ########################################### [ 14%]

2:pdsh ########################################### [ 29%]

3:pdsh-debuginfo ########################################### [ 43%]

4:pdsh-mod-dshgroup ########################################### [ 57%]

5:pdsh-mod-machines ########################################### [ 71%]

6:pdsh-mod-netgroup ########################################### [ 86%]

7:pdsh-rcmd-exec ########################################### [100%]

[root@test ~]# rpm -ivh /usr/src/redhat/RPMS/i386/pdsh-*

Preparing… ##### ( 14%)

########################################### [100%]

1:pdsh-rcmd-ssh ########################################### [ 14%]

2:pdsh ########################################### [ 29%]

3:pdsh-debuginfo ########################################### [ 43%]

4:pdsh-mod-dshgroup ########################################### [ 57%]

5:pdsh-mod-machines ########################################### [ 71%]

6:pdsh-mod-netgroup ########################################### [ 86%]

7:pdsh-rcmd-exec ########################################### [100%]

[root@ nworkhorse i386]# rpm -ivh pdsh-*

Preparing… ########################################### [100%]

1:pdsh-rcmd-ssh ########################################### [ 14%]

2:pdsh ########################################### [ 29%]

3:pdsh-debuginfo ########################################### [ 43%]

4:pdsh-mod-dshgroup ########################################### [ 57%]

5:pdsh-mod-machines ########################################### [ 71%]

6:pdsh-mod-netgroup ########################################### [ 86%]

7:pdsh-rcmd-exec ########################################### [100%]

NFS Setup

[root@hpcc1 .ssh]# vi /etc/exports
/cluster *(rw,sync)

[root@hpcc1 .ssh]# pdsh -a mkdir /cluster

failed to install module options for “misc/dshgroup”

[root@hpcc1 .ssh]#

[root@hpcc1 .ssh]#

[root@hpcc1 .ssh]# /etc/init.d/nfs restart

Shutting down NFS mountd: [FAILED]

Shutting down NFS daemon: [FAILED]

Shutting down NFS quotas: [FAILED]

Shutting down NFS services: [ OK ]

Starting NFS services: [ OK ]

Starting NFS quotas: [ OK ]

Starting NFS daemon: [ OK ]

Starting NFS mountd: [ OK ]

[root@hpcc1 .ssh]#

[root@hpcc2 .ssh]# mount -t nfs hpcc1:/cluster/ /cluster/

[root@hpcc2 .ssh]#

[root@hpcc2 .ssh]#

[root@hpcc2 .ssh]# df -

df: `-‘: No such file or directory

[root@hpcc2 .ssh]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 29G 5.4G 22G 20% /

tmpfs 1.7G 0 1.7G 0% /dev/shm

hpcc1:/cluster/ 19G 6.3G 12G 35% /cluster

[root@hpcc2 .ssh]#

Node 1

[root@hpcc1 /]# groupadd -g 604 power

[root@hpcc1 /]# useradd -u 700 -g 604 -d /cluster/power hpower

Node 2

[root@hpcc2 ~]# groupadd -g 604 power

[root@hpcc2 ~]# useradd -u 700 -g 604 -d /cluster/power hpower

useradd: warning: the home directory already exists.

Not copying any file from skel directory into it.

[root@hpcc2 ~]#

Node 3

[root@test]# mount -t nfs hpcc1:/cluster/ /cluster/

[root@test ~]# mkdir /cluster

[root@test ~]# groupadd -g 604 power

[root@test ~]# useradd -u 700 -g 604 -d /cluster/power hpower

useradd: warning: the home directory already exists.

Not copying any file from skel directory into it.

Node 4

[root@nworkhorse ~]# mkdir /cluster

[root@nworkhorse ~]# mount -t nfs hpcc1:/cluster/ /cluster/

[root@nworkhorse ~]# groupadd -g 604 power

[root@nworkhorse ~]# useradd -u 700 -g 604 -d /cluster/power hpower

useradd: warning: the home directory already exists.

Not copying any file from skel directory into it.

Node 1

[hpower@hpcc1 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/cluster/power/.ssh/id_dsa):

Created directory ‘/cluster/power/.ssh’.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /cluster/power/.ssh/id_dsa.

Your public key has been saved in /cluster/power/.ssh/id_dsa.pub.

The key fingerprint is:

79:d4:a0:4e:2e:e5:b1:05:5e:9c:93:56:06:0f:d4:1f hpower@hpcc1.isb.ots.com

[hpower@hpcc1 ~]$

[hpower@hpcc1 ~]$

[hpower@hpcc1 ~]$ ls

[hpower@hpcc1 ~]$ cd .ssh/

[hpower@hpcc1 .ssh]$ ls

id_dsa id_dsa.pub

[hpower@hpcc1 .ssh]$ vi authorized

[hpower@hpcc1 .ssh]$ vi id_dsa.pub

[hpower@hpcc1 .ssh]$ cat id_dsa.pub > authorized_keys

Node 2

[hpower@hpcc2 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/cluster/power/.ssh/id_dsa):

/cluster/power/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /cluster/power/.ssh/id_dsa.

Your public key has been saved in /cluster/power/.ssh/id_dsa.pub.

The key fingerprint is:

85:5a:0f:c2:d6:21:6e:55:38:24:5d:60:02:28:16:7b hpower@hpcc2.isb.ots.com

[hpower@hpcc2 ~]$

[hpower@hpcc2 ~]$ cd .ssh/

[hpower@hpcc2 .ssh]$ cat id_dsa.pub » authorized_keys

Node 3

[hpower@test ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/cluster/power/.ssh/id_dsa):

/cluster/power/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /cluster/power/.ssh/id_dsa.

Your public key has been saved in /cluster/power/.ssh/id_dsa.pub.

The key fingerprint is:

82:d9:8e:34:48:a7:f8:32:21:8d:e6:17:3d:2b:dd:21 hpower@test.isb.com

[hpower@test .ssh]$ cat id_dsa.pub » authorized_keys

Node 4

[hpower@nworkhorse ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/cluster/power/.ssh/id_dsa):

/cluster/power/.ssh/id_dsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /cluster/power/.ssh/id_dsa.

Your public key has been saved in /cluster/power/.ssh/id_dsa.pub.

The key fingerprint is:

c1:a6:01:ec:1e:a6:e9:8b:5e:89:2d:33:06:8b:24:cb hpower@nworkhorse.isb.ots.com

[hpower@nworkhorse ~]$ cat id_dsa.pub » authorized_keys

cat: id_dsa.pub: No such file or directory

[hpower@nworkhorse ~]$ cd .ssh/

[hpower@nworkhorse .ssh]$ cat id_dsa.pub » authorized_keys

[hpower@hpcc1 .ssh]$ chmod 644 authorized_keys

[hpower@hpcc1 .ssh]$ ssh hpcc2

The authenticity of host ‘hpcc2 (192.168.1.49)’ can’t be established.

RSA key fingerprint is 99:6f:8d:b7:8b:89:ae:63:cf:e8:3f:3e:60:d1:bb:c3.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘hpcc2,192.168.1.49’ (RSA) to the list of known hosts.

[hpower@hpcc2 ~]$ logout

Connection to hpcc2 closed.

[hpower@hpcc1 .ssh]$ ssh hpcc1

Last login: Sat Jun 6 14:25:48 2009 from hpcc2.isb.ots.com

[hpower@hpcc1 ~]$ logout

[hpower@hpcc2 .ssh]$ ssh hpcc1

The authenticity of host ‘hpcc1 (192.168.1.180)’ can’t be established.

RSA key fingerprint is a2:04:92:48:b8:58:56:b7:24:9d:e5:da:30:75:92:b6.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘hpcc1,192.168.1.180’ (RSA) to the list of known hosts.

[hpower@hpcc1 ~]$ logout

Connection to hpcc1 closed.

[hpower@hpcc2 .ssh]$ ssh hpcc2

Last login: Sat Jun 6 14:25:41 2009 from hpcc1.isb.ots.com

[hpower@hpcc2 ~]$ logout

[root@hpcc1 ~]# cp mpich2-1.0.8p1.tar.gz /cluster/

[root@hpcc1 ~]#

[root@hpcc1 /]# chown hpower.power /cluster/ -R

hpower@hpcc1 ~]$ cd ..

[hpower@hpcc1 cluster]$ ls

mpich2-1.0.8p1.tar.gz power

[hpower@hpcc1 cluster]$ tar zxvf mpich2-1.0.8p1.tar.gz

mpich2-1.0.8p1/

mpich2-1.0.8p1/mpich2.def

mpich2-1.0.8p1/test/

mpich2-1.0.8p1/test/.codingcheck

[hpower@hpcc1 mpich2-1.0.8p1]$ ./configure –prefix=/cluster/mpich2

config.status: executing default-3 commands

Configuration completed.

If done with out any error then

[hpower@hpcc1 mpich2-1.0.8p1]$ make

[hpower@hpcc1 mpich2-1.0.8p1]$ make install

[hpower@hpcc1 ~]$ vi .bash_profile

.bash_profile

Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

User specific environment and startup programs

PATH=$PATH:$HOME/bin:/cluster/mpich2/bin

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cluster/mpich2/lib

#PATH=$PATH:$HOME/bin

export PATH

export LD_LIBRARY_PATH

[mpiuser@node2 ~]$ source .bash_profile

[mpiuser@node1 ~]$ echo $PATH

/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/cluster/mpiuser/bin:/cluster/mpiuser/bin:/cluster/mpich2/bin

Create new file :

[hpower@hpcc1 ~]$ vi mpd.hosts

hpcc1

hpcc2

Create new file :

[hpower@hpcc1 ~]$ vi mpd.hosts

secretword=asim

[mpiuser@node1 ~]$ chmod 600 .mpd.conf

run

[hpower@hpcc1 ~]$ mpd &

[hpower@hpcc1 ~]$ mpd &

[1] 13499

[hpower@hpcc1 ~]$ mpdtrace

hpcc1

[hpower@hpcc1 ~]$ mpdtrace

hpcc1

[hpower@hpcc1 ~]$ mpdallexit

[hpower@hpcc1 ~]$ mpdboot -n 2

[hpower@hpcc1 ~]$ mpdtrace

hpcc1

hpcc2

[hpower@hpcc1 ~]$

Both nodes added Good

For Checkup

[hpower@hpcc1 ~]$ mpdboot -n 2 –chkuponly

checking hpcc2

there are 2 hosts up (counting local)

Now let’s start some becnhmarking

[hpower@hpcc1 cluster]$ cd mpich2-1.0.8p1/examples/

cpi file already complied and executeable

-rw-r–r– 1 hpower power 678 Nov 3 2007 child.c

-rwxr-xr-x 1 hpower power 576450 Jun 6 14:34 cpi

-rw-r–r– 1 hpower power 1515 Nov 3 2007 cpi.c

-rw-r–r– 1 hpower power 1964 Jun 6 14:34 cpi.o

-rw-r–r– 1 hpower power 4469 Nov 3 2007 cpi.vcproj

drwxr-xr-x 2 hpower power 4096 Jun 6 14:31 cxx

drwxr-xr-x 2 hpower power 4096 Mar 27 01:40 developers

-rw-r–r– 1 hpower power 10446 Nov 3 2007 examples.sln

drwxr-xr-x 2 hpower power 4096 Jun 6 14:31 f77

drwxr-xr-x 2 hpower power 4096 Jun 6 14:31 f90

-rw-r–r– 1 hpower power 455 Nov 3 2007 hellow.c

-rw-r–r– 1 hpower power 1892 Nov 3 2007 icpi.c

-rw-r–r– 1 hpower power 6802 Jun 6 14:31 Makefile

-rw-r–r– 1 hpower power 6767 Mar 27 01:40 Makefile.in

-rw-r–r– 1 hpower power 1490 Mar 12 2008 Makefile.sm

drwxr-xr-x 2 hpower power 4096 Mar 27 01:39 mpiexec

-rw-r–r– 1 hpower power 1049 Nov 3 2007 parent.c

-rw-r–r– 1 hpower power 46399 Nov 3 2007 pmandel.c

-rw-r–r– 1 hpower power 47798 Nov 3 2007 pmandel_fence.c

-rw-r–r– 1 hpower power 4522 Nov 3 2007 pmandel_fence.vcproj

-rw-r–r– 1 hpower power 45576 Nov 3 2007 pmandel_service.c

-rw-r–r– 1 hpower power 4532 Nov 3 2007 pmandel_service.vcproj

-rw-r–r– 1 hpower power 47510 Nov 3 2007 pmandel_spaserv.c

-rw-r–r– 1 hpower power 4510 Nov 3 2007 pmandel_spaserv.vcproj

-rw-r–r– 1 hpower power 49858 Nov 3 2007 pmandel_spawn.c

-rw-r–r– 1 hpower power 4522 Nov 3 2007 pmandel_spawn.vcproj

-rw-r–r– 1 hpower power 4492 Nov 3 2007 pmandel.vcproj

-rw-r–r– 1 hpower power 2175 Nov 3 2007 README

-rw-r–r– 1 hpower power 1116 Nov 3 2007 spawn_merge_child1.c

-rw-r–r– 1 hpower power 694 Nov 3 2007 spawn_merge_child2.c

-rw-r–r– 1 hpower power 1795 Nov 3 2007 spawn_merge_parent.c

-rw-r–r– 1 hpower power 3554 Nov 3 2007 spawntest.vcproj

-rw-r–r– 1 hpower power 1642 Nov 3 2007 srtest.c

drwxr-xr-x 7 hpower power 4096 Mar 27 01:40 windows

[hpower@hpcc1 ~]$ cd -

/cluster/mpich2-1.0.8p1/examples

[hpower@hpcc1 examples]$ mpiexec -n 1 ./cpi

Lamnodes Failed!

Check if you had booted lam before calling mpiexec else use -machinefile to pass host file to mpiexec

LAM Error but i m not using LAM ………..

After 20 mins fiding finally i found the reason mpiexec was running from /usr/bin/mpiexec

[hpower@hpcc1 ~]$ which mpiexec

/usr/bin/mpiexec

If run this using absolute path it is working. Means there is some problem with our .bash_profile

[hpower@hpcc1 ~]$ /cluster/mpich2/bin/mpiexec -n 1 /cluster/mpich2-1.0.8p1/examples/cpi

Process 0 of 1 is on hpcc1.isb.ots.com

pi is approximately 3.1415926544231341, Error is 0.0000000008333410

wall clock time = 0.000317

I found only one solution to solve

[root@hpcc1 ~]# mv /usr/bin/mpiexec /tmp/ :)

Its Working Now

[hpower@hpcc1 ~]$ which mpiexec

/cluster/mpich2/bin/mpiexec

[hpower@hpcc1 ~]$

[hpower@hpcc1 ~]$ mpiexec -n 1 /cluster/mpich2-1.0.8p1/examples/cpi

Process 0 of 1 is on hpcc1.isb.ots.com

pi is approximately 3.1415926544231341, Error is 0.0000000008333410

wall clock time = 0.000297

[hpower@hpcc1 ~]$ mpiexec -n 2 /cluster/mpich2-1.0.8p1/examples/cpi

Process 0 of 2 is on hpcc1.isb.ots.com

Process 1 of 2 is on hpcc2.isb.ots.com

pi is approximately 3.1415926544231318, Error is 0.0000000008333387

wall clock time = 0.001234

[hpower@hpcc1 ~]$

There is an other program available for testing lets compile it first

/cluster/mpich2-1.0.8p1/examples

[hpower@hpcc1 examples]$ mpicc -o icpi icpi.c

Now execute it and also open top -c to check the processes.

First we will run with single node

[hpower@hpcc1 examples]$ mpiexec -n 1 ./icpi

Enter the number of intervals: (0 quits) 1000000000

pi is approximately 3.1415926535921401, Error is 0.0000000000023470

wall clock time = 22.537397

Enter the number of intervals: (0 quits)

top -c out put :

[root@hpcc1 ~]# top -c

top - 15:53:08 up 1:57, 3 users, load average: 0.21, 0.11, 0.03

Tasks: 188 total, 2 running, 186 sleeping, 0 stopped, 0 zombie

Cpu(s): 12.5%us, 0.0%sy, 0.0%ni, 87.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 3368196k total, 936760k used, 2431436k free, 140504k buffers

Swap: 12289716k total, 0k used, 12289716k free, 634932k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14289 hpower 25 0 2332 812 676 R 100 0.0 0:07.80./icpi

1 root 15 0 2032 640 552 S 0 0.0 0:01.82 init [5]

Lets run with both nodes

[hpower@hpcc1 examples]$ mpiexec -n 2 ./icpi

Enter the number of intervals: (0 quits) 1000000000

pi is approximately 3.1415926535905170, Error is 0.0000000000007239

wall clock time = 11.285166

Enter the number of intervals: (0 quits)

HPCC2 top -c output

top - 15:56:47 up 2:01, 5 users, load average: 0.27, 0.08, 0.06

Tasks: 205 total, 2 running, 203 sleeping, 0 stopped, 0 zombie

Cpu(s): 12.5%us, 0.0%sy, 0.0%ni, 87.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 3368120k total, 1858180k used, 1509940k free, 197464k buffers

Swap: 13309844k total, 0k used, 13309844k free, 1488212k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

4615 hpower 25 0 2356 812 676 R 100 0.0 0:10.26 ./icpi

4593 root 15 0 2320 1080 800 R 0 0.0 0:01.06 top -c

HPCC1 top -c Out put

[root@hpcc1 ~]# top -c

top - 15:56:46 up 2:01, 3 users, load average: 0.14, 0.10, 0.04

Tasks: 188 total, 2 running, 186 sleeping, 0 stopped, 0 zombie

Cpu(s): 12.5%us, 0.0%sy, 0.0%ni, 87.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 3368196k total, 941472k used, 2426724k free, 140656k buffers

Swap: 12289716k total, 0k used, 12289716k free, 634876k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14312 hpower 25 0 2328 848 712 R 100 0.0 0:08.84 ./icpi

1 root 15 0 2032 640 552 S 0 0.0 0:01.82 init [5]

Now I am going to add nodes 3 on /etc/mpd.hosts

[hpower@hpcc1 ~]$ vi mpd.hosts

hpcc1

hpcc2

test

[hpower@hpcc1 ~]$ mpdallexit

[hpower@hpcc1 ~]$ mpdboot -n 3

[hpower@hpcc1 ~]$ mpdtrace

hpcc1

test

hpcc2

All 3 nodes addes successfully

Now lets run scripts again for benchmarking

[hpower@hpcc1 ~]$ cd /cluster/mpich2-1.0.8p1/examples/

[hpower@hpcc1 examples]$ mpiexec -n 2 ./icpi

Enter the number of intervals: (0 quits) 1000000000

pi is approximately 3.1415926535905170, Error is 0.0000000000007239

wall clock time = 43.697841

[hpower@hpcc1 examples]$ mpiexec -n 3 ./icpi

Enter the number of intervals: (0 quits) 1000000000

pi is approximately 3.1415926535899761, Error is 0.0000000000001830

wall clock time = 29.853977

Hmmmm better then two nodes add another 0 :) It will take some time .

Add node 4

there are not enough hosts on which to start all processes

[hpower@hpcc1 ~]$ vi mpd.hosts

[hpower@hpcc1 ~]$ mpdboot -n 4

[hpower@hpcc1 ~]$ mpdtrace

hpcc1

test

nworkhorse

hpcc2

[hpower@hpcc1 examples]$ mpiexec -n 4 ./icpi

Enter the number of intervals: (0 quits) 1000000000

pi is approximately 3.1415926535896128, Error is 0.0000000000001803

wall clock time = 21.904199

Lets Test GoTo Blast.

Download Location

http://www.tacc.utexas.edu/resources/software/login/gotoblas/GotoBLAS-1.26.tar.gz

[root@hpcc1 ~]# cp GotoBLAS-1.26.tar.gz /cluster/

[root@hpcc1 ~]# cd /cluster/

[root@hpcc1 cluster]# ls

GotoBLAS-1.26.tar.gz mpich2 mpich2-1.0.8p1 mpich2-1.0.8p1.tar.gz power

[root@hpcc1 cluster]# tar zxvf GotoBLAS-1.26.tar.gz

[root@hpcc1 cluster]# su - hpower

[hpower@hpcc1 cluster]$ cd GotoBLAS

[hpower@hpcc1 GotoBLAS]$ ./quickbuild.32bit

[hpower@hpcc1 GotoBLAS]$ make

It takes some time

Now download hpl

Download Link

http://www.netlib.org/benchmark/hpl/hpl-2.0.tar.gz

http://www.netlib.org/benchmark/hpl/hpl.tgz

root@hpcc1 cluster]# cd

[root@hpcc1 ~]# ls

@ Desktop hpl.tgz ipscan-3.0_beta4-1.i386.rpm mpich2-1.0.8p1.tar.gz snapshot1.jpg snapshot4.jpg

abc.sh GotoBLAS-1.26.tar.gz install.log login.php?product=server20 pdsh-2.10-4.src.rpm snapshot2.jpg tm

anaconda-ks.cfg hpl-2.0.tar.gz install.log.syslog mac.txt

[hpower@hpcc1 cluster]$ ls

GotoBLAS GotoBLAS-1.26.tar.gz hpl-2.0.tar.gz hpl.tgz mpich2 mpich2-1.0.8p1 mpich2-1.0.8p1.tar.gz power

[hpower@hpcc1 cluster]$ tar zxf hpl-2.0.tar.gz

[hpower@hpcc1 cluster]$ tar zxf hpl.tgz

[hpower@hpcc1 hpl]$ cp setup/Make.Linux_PII_FBLAS_gm .

First check GCC version/Path

[hpower@hpcc1 hpl]$ gcc -v

Using built-in specs.

Target: i386-redhat-linux

Configured with: ../configure –prefix=/usr –mandir=/usr/share/man –infodir=/usr/share/info –enable-shared –enable-threads=posix –enable-checking=release –with-system-zlib –enable-__cxa_atexit –disable-libunwind-exceptions –enable-libgcj-multifile –enable-languages=c,c++,objc,obj-c++,java,fortran,ada –enable-java-awt=gtk –disable-dssi –enable-plugin –with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre –with-cpu=generic –host=i386-redhat-linux

Thread model: posix

/usr/lib/gcc/i386-redhat-linux/4.1.1/

Now edit Make.Linux_PII_FBLAS_gm file

[hpower@hpcc1 hpl]$ vi Make.Linux_PII_FBLAS_gm

Edit the following lines :

ARCH = Linux_PII_FBLAS_gm

TOPdir = $(HOME)/hpl

INCdir = $(TOPdir)/include

BINdir = $(TOPdir)/bin/$(ARCH)

LIBdir = $(TOPdir)/lib/$(ARCH)

#

HPLlib = $(LIBdir)/libhpl.a

LAdir = /cluster/GotoBLAS

LAlib = $(LAdir)/libgoto.a -1m -L/usr/lib/gcc/i386-redhat-linux/4.1.1/

CC = mpicc

CCFLAGS = $(HPL_DEFS) -O3

LINKER = mpicc

Ok Done now compile it .

[hpower@hpcc1 hpl]$ make arch=Linux_PII_FBLAS_gm

[hpower@test hpl]$ cd /cluster/power/hpl/bin/Linux_PII_FBLAS_gm/

[hpower@test Linux_PII_FBLAS_gm]$ ls

HPL.dat xhpl

[hpower@test Linux_PII_FBLAS_gm]$ cp HPL.dat HPL.dat.orig

MY HPL parameter First testing with 2 nodes

Innovative Computing Laboratory, University of Tennessee

HPL.out output file name (if any)

6 device out (6=stdout,7=stderr,file)

2 # of problems sizes (N)

7000 Ns

4 # of NBs

100 NBs

0 PMAP process mapping (0=Row-,1=Column-major)

3 # of process grids (P x Q)

1 Ps

2 Qs

16.0 threshold

3 # of panel fact

0 1 2 PFACTs (0=left, 1=Crout, 2=Right)

2 # of recursive stopping criterium

2 4 NBMINs (>= 1)

1 # of panels in recursion

2 NDIVs

3 # of recursive panel fact.

0 1 2 RFACTs (0=left, 1=Crout, 2=Right)

1 # of broadcast

0 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)

1 # of lookahead depth

0 DEPTHs (>=0)

2 SWAP (0=bin-exch,1=long,2=mix)

64 swapping threshold

0 L1 in (0=transposed,1=no-transposed) form

0 U in (0=transposed,1=no-transposed) form

1 Equilibration (0=no,1=yes)

8 memory alignment in double (> 0)

~

~

[hpower@hpcc1 Linux_PII_FBLAS_gm]$ mpiexec -n 2 ./xhpl

The following parameter values will be used:

N : 6000

NB : 100

PMAP : Row-major process mapping

P : 1

Q : 2

PFACT : Left Crout Right

NBMIN : 2 4

NDIV : 2

RFACT : Left Crout Right

BCAST : 1ring

DEPTH : 0

SWAP : Mix (threshold = 64)

L1 : transposed form

U : transposed form

EQUIL : yes

ALIGN : 8 double precision words

top -c on node 1 (hpcc1) Procssor speed 3.0 dual core

Cpu(s): 6.2%us, 1.5%sy, 0.0%ni, 92.1%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st

Mem: 3368196k total, 1229388k used, 2138808k free, 215208k buffers

Swap: 12289716k total, 0k used, 12289716k free, 695944k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

16856 hpower 15 0 172m 141m 1016 S 62 4.3 0:55.41 ./xhpl

top -c on node 3 (test ) 1.7 MHZ PC class machine

Cpu(s): 78.1%us, 1.3%sy, 0.0%ni, 19.6%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 443392k total, 437712k used, 5680k free, 21312k buffers

Swap: 1052248k total, 0k used, 1052248k free, 189308k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14055 hpower 25 0 175m 144m 960 R 78.6 33.4 0:59.60 ./xhpl

  • The matrix A is randomly generated for each test.

  • The following scaled residual checks will be computed:

1) ||Ax-b||oo / ( eps * ||A||1 * N )

2) ||Ax-b||oo / ( eps * ||A||1 * ||x||1 )

3) ||Ax-b||oo / ( eps * ||A||oo * ||x||oo)

  • The relative machine precision (eps) is taken to be 1.110223e-16

  • Computational tests pass if scaled residuals are less than 16.0

============================================================================

T/V N NB P Q Time Gflops


WR00L2L2 6000 100 1 2 46.28 3.113e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0034911 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0072807 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0014339 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2L4 6000 100 1 2 49.01 2.939e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0038909 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0081145 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0015981 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2C2 6000 100 1 2 45.78 3.146e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0034911 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0072807 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0014339 ...... PASSED

Again RUN now incrase the size of NS.

The following parameter values will be used:

N : 7000

NB : 100

PMAP : Row-major process mapping

P : 1

Q : 2

PFACT : Left Crout Right

NBMIN : 2 4

NDIV : 2

RFACT : Left Crout Right

BCAST : 1ring

DEPTH : 0

SWAP : Mix (threshold = 64)

L1 : transposed form

U : transposed form

EQUIL : yes

ALIGN : 8 double precision words

  • Computational tests pass if scaled residuals are less than 16.0

============================================================================

T/V N NB P Q Time Gflops


WR00L2L2 7000 100 1 2 72.10 3.173e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.1808081 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091439 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0016514 ...... PASSED

With 4 node

Cpu(s): 2.9%us, 0.7%sy, 0.0%ni, 96.3%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st

Mem: 3368196k total, 1120648k used, 2247548k free, 215220k buffers

Swap: 12289716k total, 0k used, 12289716k free, 695976k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

17007 hpower 15 0 68892 34m 1016 S 28 1.0 0:28.31 ./xhpl

Cpu(s): 62.2%us, 20.4%sy, 0.0%ni, 8.6%id, 8.6%wa, 0.0%hi, 0.3%si, 0.0%st

Mem: 443392k total, 175608k used, 267784k free, 2532k buffers

Swap: 1052248k total, 41448k used, 1010800k free, 86836k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14144 hpower 15 0 70652 36m 960 R 63.1 8.5 0:10.83 ./xhpl

14066 root 15 0 2212 948 752 R 0.3 0.2 0:01.75 top -c

Cpu(s): 36.0%us, 10.3%sy, 0.0%ni, 50.0%id, 0.0%wa, 1.7%hi, 2.0%si, 0.0%st

Mem: 969896k total, 602732k used, 367164k free, 31900k buffers

Swap: 2096408k total, 0k used, 2096408k free, 374868k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

4803 hpower 16 0 70416 36m 960 R 46.5 3.9 0:30.35 ./xhpl

============================================================================

HPLinpack 1.0a – High-Performance Linpack benchmark – January 20, 2004

Written by A. Petitet and R. Clint Whaley, Innovative Computing Labs., UTK

============================================================================

An explanation of the input/output parameters follows:

T/V : Wall time / encoded variant.

N : The order of the coefficient matrix A.

NB : The partitioning blocking factor.

P : The number of process rows.

Q : The number of process columns.

Time : Time in seconds to solve the linear system.

Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N : 4000

NB : 100

PMAP : Row-major process mapping

P : 2

Q : 2

PFACT : Left Crout Right

NBMIN : 2 4

NDIV : 2

RFACT : Left Crout Right

BCAST : 1ring

DEPTH : 0

SWAP : Mix (threshold = 64)

L1 : transposed form

U : transposed form

EQUIL : yes

ALIGN : 8 double precision words


  • The matrix A is randomly generated for each test.

  • The following scaled residual checks will be computed:

1) ||Ax-b||oo / ( eps * ||A||1 * N )

2) ||Ax-b||oo / ( eps * ||A||1 * ||x||1 )

3) ||Ax-b||oo / ( eps * ||A||oo * ||x||oo)

  • The relative machine precision (eps) is taken to be 1.110223e-16

  • Computational tests pass if scaled residuals are less than 16.0

============================================================================

T/V N NB P Q Time Gflops


WR00L2L2 4000 100 2 2 46.16 9.248e-01


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2L4 4000 100 2 2 18.48 2.310e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2C2 4000 100 2 2 21.46 1.990e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2C4 4000 100 2 2 18.39 2.322e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2R2 4000 100 2 2 18.68 2.285e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00L2R4 4000 100 2 2 17.58 2.428e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098668 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091118 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020710 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2L2 4000 100 2 2 19.05 2.241e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2L4 4000 100 2 2 17.81 2.398e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2C2 4000 100 2 2 17.51 2.438e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2C4 4000 100 2 2 18.25 2.340e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2R2 4000 100 2 2 18.62 2.293e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00C2R4 4000 100 2 2 17.84 2.393e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098668 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091118 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020710 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2L2 4000 100 2 2 18.31 2.331e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2L4 4000 100 2 2 17.48 2.442e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2C2 4000 100 2 2 18.03 2.368e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2C4 4000 100 2 2 18.90 2.259e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0086843 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0080198 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0018228 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2R2 4000 100 2 2 18.64 2.290e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098900 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091332 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020759 ...... PASSED

============================================================================

T/V N NB P Q Time Gflops


WR00R2R4 4000 100 2 2 18.59 2.296e+00


||Ax-b||oo / ( eps * ||A||1 * N ) = 0.0098668 ...... PASSED

||Ax-b||oo / ( eps * ||A||1 * ||x||1 ) = 0.0091118 ...... PASSED

||Ax-b||oo / ( eps * ||A||oo * ||x||oo) = 0.0020710 ...... PASSED

============================================================================

Finished 18 tests with the following results:

18 tests completed and PASSED` residual checks,

0 tests completed and failed residual checks,

0 tests skipped because of illegal input values.


End of Tests.