From: Maestas, Christopher Daniel (cdmaest_at_[hidden])
Date: 2007-05-18 15:58:41
Much better thanks!
---
$ env | grep OMPI
OMPI_MCA_rmaps_base_schedule_policy=node
OMPI_MCA_pml=cm
OMPI_MCA_btl=^openib
OMPI_MCA_oob_tcp_include=eth0
OMPI_MCA_mpi_keep_hostnames=1
$ mpiexec -pernode -np 2 IMB-MPI1 SendRecv
#---------------------------------------------------
# Intel (R) MPI Benchmark Suite V2.3, MPI-1 part
#---------------------------------------------------
# Date : Fri May 18 13:56:36 2007
# Machine : x86_64# System : Linux
# Release : 2.6.9-42.0.10.EL_lustre.1.4.10smp
# Version : #1 SMP Tue May 15 19:56:35 MDT 2007
#
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
#-----------------------------------------------------------------------
------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------
------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
Mbytes/sec
0 1000 4.06 4.06 4.06
0.00
1 1000 4.22 4.22 4.22
0.45
2 1000 4.40 4.40 4.40
0.87
4 1000 4.36 4.36 4.36
1.75
8 1000 4.35 4.35 4.35
3.50
16 1000 4.42 4.42 4.42
6.90
32 1000 4.59 4.59 4.59
13.29
64 1000 4.70 4.71 4.71
25.93
128 1000 5.17 5.17 5.17
47.18
256 1000 7.24 7.25 7.24
67.38
512 1000 8.90 8.91 8.91
109.63
1024 1000 12.34 12.35 12.34
158.21
2048 1000 15.76 15.78 15.77
247.57
4096 1000 21.24 21.25 21.24
367.73
8192 1000 35.53 35.54 35.53
439.71
16384 1000 60.57 60.58 60.57
515.89
32768 1000 105.18 105.19 105.18
594.15
65536 640 197.25 197.25 197.25
633.70
131072 320 365.81 365.82 365.81
683.40
262144 160 696.87 696.88 696.87
717.49
524288 80 1376.61 1376.72 1376.67
726.36
1048576 40 2688.57 2691.88 2690.23
742.98
2097152 20 5311.70 5312.20 5311.95
752.98
4194304 10 10557.51 10558.80 10558.15
757.66
> -----Original Message-----
> From: users-bounces_at_[hidden]
> [mailto:users-bounces_at_[hidden]] On Behalf Of gshipman
> Sent: Friday, May 18, 2007 1:48 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Intel MPI SendRcv Performance using
> MX 1.1.7 comparison of Open MPI 1.2.2 and MPICH-MX 1.2.7..4
>
> Can you try adding the following param:
>
> OMPI_MCA_pml=cm
>
> and report the results?
>
> Thanks,
>
> Galen
>
>
> On May 18, 2007, at 1:15 PM, Maestas, Christopher Daniel wrote:
>
> > Hello,
> >
> > I was wondering why we would see ~ 100MB/s difference
> between mpich-mx
> > and Open MPI with SendRecv from the Intel MPI benchmarks. Maybe I'm
> > missing turning something on?
> >
> > The hardware is:
> > ---
> > # mx_info -q
> > MX Version: 1.1.7
> > MX Build:
> root_at_tocc1:/projects/global/SOURCES/myricom/mx-1.1.7 Fri May
> > 11 15:32:14 MDT 2007
> > 1 Myrinet board installed.
> > The MX driver is configured to support up to 4 instances and 1024
> > nodes.
> > ===================================================================
> > Instance #0: 333.2 MHz LANai, 133.3 MHz PCI bus, 4 MB SRAM
> > Status: Running, P0: Link up, P1: Link up
> > MAC Address: 00:60:dd:48:ba:ae
> > Product code: M3F2-PCIXE-4
> > ---
> >
> > --- Open MPI 1.2.2 ---
> > $ cat
> > /apps/x86_64/mpi/openmpi/intel-9.1-f040-c045/openmpi-1.2.2_mx/
> > BUILD_ENV
> > # Build Environment:
> > USE="doc icc modules mx torque"
> > COMPILER="intel-9.1-f040-c045"
> > CC="icc"
> > CXX="icpc"
> > CLINKER="icc"
> > FC="ifort"
> > F77="ifort"
> > CFLAGS=" -O3 -pipe"
> > CXXFLAGS=" -O3 -pipe"
> > FFLAGS=" -O3"
> > MODULE_DEST="/apps/modules/modulefiles/mpi"
> > MODULE_FILE="openmpi-1.2.2_mx_intel-9.1-f040-c045"
> > INSTALL_DEST="/apps/x86_64/mpi/openmpi/intel-9.1-f040-c045/
> > openmpi-1.2.2
> > _mx"
> > CONF_FLAGS=" --with-mx=/opt/mx --with-tm=/apps/torque"
> > $ env | grep OMPI
> > OMPI_MCA_rmaps_base_schedule_policy=node
> > OMPI_MCA_btl=^openib
> > OMPI_MCA_oob_tcp_include=eth0
> > OMPI_MCA_mpi_keep_hostnames=1
> > $ mpiexec -pernode -np 2 IMB-MPI1 SendRecv
> > #---------------------------------------------------
> > # Intel (R) MPI Benchmark Suite V2.3, MPI-1 part
> > #---------------------------------------------------
> > # Date : Fri May 18 13:11:20 2007
> > # Machine : x86_64# System : Linux
> > # Release : 2.6.9-42.0.10.EL_lustre.1.4.10smp
> > # Version : #1 SMP Tue May 15 19:56:35 MDT 2007
> >
> > #
> > # Minimum message length in bytes: 0
> > # Maximum message length in bytes: 4194304
> > #
> > # MPI_Datatype : MPI_BYTE
> > # MPI_Datatype for reductions : MPI_FLOAT
> > # MPI_Op : MPI_SUM
> > #
> > #
> >
> > # List of Benchmarks to run:
> >
> > # Sendrecv
> >
> >
> #-------------------------------------------------------------
> --------
> > --
> > ------
> > # Benchmarking Sendrecv
> > # #processes = 2
> >
> #-------------------------------------------------------------
> --------
> > --
> > ------
> > #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
> > Mbytes/sec
> > 0 1000 4.38 4.38 4.38
> > 0.00
> > 1 1000 6.03 6.03 6.03
> > 0.32
> > 2 1000 6.02 6.03 6.02
> > 0.63
> > 4 1000 6.10 6.10 6.10
> > 1.25
> > 8 1000 6.27 6.27 6.27
> > 2.43
> > 16 1000 6.23 6.23 6.23
> > 4.90
> > 32 1000 6.50 6.50 6.50
> > 9.39
> > 64 1000 6.48 6.49 6.49
> > 18.82
> > 128 1000 8.02 8.02 8.02
> > 30.45
> > 256 1000 8.69 8.69 8.69
> > 56.20
> > 512 1000 10.16 10.16 10.16
> > 96.08
> > 1024 1000 13.51 13.52 13.52
> > 144.48
> > 2048 1000 17.19 17.19 17.19
> > 227.25
> > 4096 1000 24.21 24.21 24.21
> > 322.67
> > 8192 1000 38.10 38.10 38.10
> > 410.13
> > 16384 1000 64.46 64.46 64.46
> > 484.82
> > 32768 1000 116.23 116.24 116.23
> > 537.69
> > 65536 640 210.11 210.11 210.11
> > 594.91
> > 131072 320 401.53 401.56 401.54
> > 622.58
> > 262144 160 900.95 901.01 900.98
> > 554.94
> > 524288 80 1909.00 1909.20 1909.10
> > 523.78
> > 1048576 40 3868.03 3868.40 3868.21
> > 517.01
> > 2097152 20 8316.95 8317.46 8317.20
> > 480.92
> > 4194304 10 19240.28 19241.38 19240.83
> > 415.77
> >
> > --- mpich-mx --
> > $ cat
> > /apps/x86_64/mpi/mpich-mx/intel-9.1-f040-c045/mpich-mx-1.2.7..4/
> > BUILD_EN
> > V
> > # Build Environment:
> > USE="doc icc modules romio"
> > COMPILER="intel-9.1-f040-c045"
> > CC="icc"
> > CXX="icpc"
> > CLINKER="icc"
> > FC="ifort"
> > F77="ifort"
> > CFLAGS="-O3 -pipe -xP -I/opt/mx/include"
> > CXXFLAGS="-O3 -pipe -xP -I/opt/mx/include"
> > FFLAGS="-O3 -xP"
> > MODULE_DEST="/apps/modules/modulefiles/mpi"
> > MODULE_FILE="mpich-mx-1.2.7..4_intel-9.1-f040-c045"
> > INSTALL_DEST="/apps/x86_64/mpi/mpich-mx/intel-9.1-f040-c045/mpich-
> > mx-1.2
> > .7.4"
> >
> > $ mpiexec -comm=mx -pernode -np 2 IMB-MPI1 SendRecv
> >
> > #---------------------------------------------------
> > # Intel (R) MPI Benchmark Suite V2.3, MPI-1 part
> > #---------------------------------------------------
> > # Date : Fri May 18 12:59:21 2007
> > # Machine : x86_64# System : Linux
> > # Release : 2.6.9-42.0.10.EL_lustre.1.4.10smp
> > # Version : #1 SMP Tue May 15 19:56:35 MDT 2007
> >
> > #
> > # Minimum message length in bytes: 0
> > # Maximum message length in bytes: 4194304
> > #
> > # MPI_Datatype : MPI_BYTE
> > # MPI_Datatype for reductions : MPI_FLOAT
> > # MPI_Op : MPI_SUM
> > #
> > #
> >
> > # List of Benchmarks to run:
> >
> > # Sendrecv
> >
> >
> #-------------------------------------------------------------
> --------
> > --
> > ------
> > # Benchmarking Sendrecv
> > # #processes = 2
> >
> #-------------------------------------------------------------
> --------
> > --
> > ------
> > #bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
> > Mbytes/sec
> > 0 1000 3.69 3.70 3.69
> > 0.00
> > 1 1000 3.84 3.84 3.84
> > 0.50
> > 2 1000 3.83 3.83 3.83
> > 1.00
> > 4 1000 3.96 3.96 3.96
> > 1.93
> > 8 1000 4.01 4.01 4.01
> > 3.81
> > 16 1000 4.07 4.08 4.08
> > 7.49
> > 32 1000 4.19 4.19 4.19
> > 14.56
> > 64 1000 4.32 4.32 4.32
> > 28.27
> > 128 1000 4.75 4.75 4.75
> > 51.35
> > 256 1000 6.72 6.73 6.72
> > 72.58
> > 512 1000 8.32 8.32 8.32
> > 117.32
> > 1024 1000 11.79 11.81 11.80
> > 165.39
> > 2048 1000 15.36 15.38 15.37
> > 254.03
> > 4096 1000 20.64 20.65 20.65
> > 378.38
> > 8192 1000 34.89 34.90 34.90
> > 447.71
> > 16384 1000 59.81 59.83 59.82
> > 522.32
> > 32768 1000 105.03 105.06 105.04
> > 594.90
> > 65536 640 195.61 195.61 195.61
> > 639.02
> > 131072 320 363.85 363.85 363.85
> > 687.09
> > 262144 160 693.94 693.97 693.96
> > 720.49
> > 524288 80 1369.70 1369.79 1369.74
> > 730.04
> > 1048576 40 2680.43 2680.60 2680.51
> > 746.10
> > 2097152 20 5308.65 5309.50 5309.08
> > 753.37
> > 4194304 10 10551.50 10551.91 10551.70
> > 758.16
> >
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>