Subject: Re: [OMPI users] 50% performance reduction due to OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead of using Infiniband
From: Gus Correa (gus_at_[hidden])
Date: 2009-06-24 19:19:14
Hi Jim, list
1) libnuma (non-uniform memory access, IIRR) is important if
you have AMD Opteron (our case here),
for processor and memory affinity, etc.
I suppose you don't need it with Intel Xeon (pre-Nehalem, at least),
but I am not positive about this (we don't have a Xeon cluster here).
2) tm is the Torque resource manager.
If you don't use Torque/PBS you don't have it, you don't need it.
Noam for instance uses sge, and configured with --use-sge.
If you just launch mpiexec directly you don't need
to build OpenMPI with any resource manager library or support.
(But resource managers are great!)
3) It may be that --enable-static puts the wrapper flags you don't have.
I don't know. The OpenMPI developers may clarify this.
4) AFAIK, you need OFED on all nodes, at least on those
that have IB hardware, that are connected to your IB switch,
where you want to run MPI programs using IB.
I hope this helps.
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Jim Kress ORG wrote:
> Well, the whole situation is really bizarre.
>
> I just uninstalled openmpi 1.3.2 on my system. Then I installed OFED
> 1.4.1 to see if that resolves this situation.
>
> Here's what I get:
>
> [root_at_master ~]# ompi_info --config
> Configured by: root
> Configured on: Wed Jun 24 11:10:00 EDT 2009
> Configure host: master.org
> Built by: root
> Built on: Wed Jun 24 11:13:22 EDT 2009
> Built host: master.org
> C bindings: yes
> C++ bindings: yes
> Fortran77 bindings: yes (all)
> Fortran90 bindings: yes
> Fortran90 bindings size: small
> C compiler: gcc
> C compiler absolute: /usr/bin/gcc
> C char size: 1
> C bool size: 1
> C short size: 2
> C int size: 4
> C long size: 8
> C float size: 4
> C double size: 8
> C pointer size: 8
> C char align: 1
> C bool align: 1
> C int align: 4
> C float align: 4
> C double align: 8
> C++ compiler: g++
> C++ compiler absolute: /usr/bin/g++
> Fortran77 compiler: gfortran
> Fortran77 compiler abs: /usr/bin/gfortran
> Fortran90 compiler: gfortran
> Fortran90 compiler abs: /usr/bin/gfortran
> Fort integer size: 4
> Fort logical size: 4
> Fort logical value true: 1
> Fort have integer1: yes
> Fort have integer2: yes
> Fort have integer4: yes
> Fort have integer8: yes
> Fort have integer16: no
> Fort have real4: yes
> Fort have real8: yes
> Fort have real16: no
> Fort have complex8: yes
> Fort have complex16: yes
> Fort have complex32: no
> Fort integer1 size: 1
> Fort integer2 size: 2
> Fort integer4 size: 4
> Fort integer8 size: 8
> Fort integer16 size: -1
> Fort real size: 4
> Fort real4 size: 4
> Fort real8 size: 8
> Fort real16 size: -1
> Fort dbl prec size: 4
> Fort cplx size: 4
> Fort dbl cplx size: 4
> Fort cplx8 size: 8
> Fort cplx16 size: 16
> Fort cplx32 size: -1
> Fort integer align: 4
> Fort integer1 align: 1
> Fort integer2 align: 2
> Fort integer4 align: 4
> Fort integer8 align: 8
> Fort integer16 align: -1
> Fort real align: 4
> Fort real4 align: 4
> Fort real8 align: 8
> Fort real16 align: -1
> Fort dbl prec align: 4
> Fort cplx align: 4
> Fort dbl cplx align: 4
> Fort cplx8 align: 4
> Fort cplx16 align: 8
> Fort cplx32 align: -1
> C profiling: yes
> C++ profiling: yes
> Fortran77 profiling: yes
> Fortran90 profiling: yes
> C++ exceptions: no
> Thread support: posix (mpi: no, progress: no)
> Sparse Groups: no
> Build CFLAGS: -DNDEBUG -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic
> -finline-functions -fno-strict-aliasing
> -pthread
> -fvisibility=hidden
> Build CXXFLAGS: -DNDEBUG -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic
> -finline-functions -pthread
> Build FFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic
> Build FCFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic
> Build LDFLAGS: -export-dynamic
> Build LIBS: -lnsl -lutil -lm
> Wrapper extra CFLAGS: -pthread
> Wrapper extra CXXFLAGS: -pthread
> Wrapper extra FFLAGS: -pthread
> Wrapper extra FCFLAGS: -pthread
> Wrapper extra LDFLAGS:
> Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
> -ldl
> Internal debug support: no
> MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
> libltdl support: yes
> Heterogeneous support: no
> mpirun default --prefix: yes
> MPI I/O support: yes
> MPI_WTIME support: gettimeofday
> Symbol visibility support: yes
> FT Checkpoint support: no (checkpoint thread: no)
> [root_at_master ~]#
>
> so you see, even the OFED 1.4.1 installation fails to put -libverbs etc
> into openmpi.
>
> Also, I think its
>
> --enable-static
>
> that is putting the -libverbs into your openmpi. I'll try it and see
> what happens.
>
> What are libnuma and tm? Do I need to worry about them?
>
> Finally, I have forgotton what I do with all the RPMs OFED generates.
> Do I install them all on my compute nodes or just a subset?
>
> Thanks for the help.
>
> Jim
>
>
> On Wed, 2009-06-24 at 17:22 -0400, Gus Correa wrote:
>> Hi Jim
>>
>>
>> Jim Kress wrote:
>> > Noam, Gus and List,
>> >
>> > Did you statically link your openmpi when you built it? If you did (the
>> > default is NOT to do this) then that could explain the discrepancy.
>> >
>> > Jim
>>
>> No, I didn't link statically.
>>
>> Did you link statically?
>>
>> Actually, I tried to do it, and it didn't work.
>> I wouldn't get OpenMPI with IB if I tried to
>> link statically (i.e. by passing -static or equivalent to CFLAGS,
>> FFLAGS, etc).
>> When I removed the "-static" I got OpenMPI with IB.
>> I always dump the configure output (and the make output, etc) to
>> log files to check these things out after it is done.
>> I really suggest you do this, it pays off, saves time, costs nothing.
>> I don't remember exactly what symptoms I found on the log,
>> whether the log definitely said that there was no IB support,
>> or if it didn't have the right flags (-libverbs, etc) like yours.
>> However, when I suppressed the "-static" from the compiler flags
>> then I've got all the IB goodies! :)
>>
>> Here is how I run configure (CFLAGS etc only have optimization flags,
>> no "-static"):
>>
>> ./configure \
>> --prefix=/my/directory \
>> --with-libnuma=/usr \
>> --with-tm=/usr \
>> --with-openib=/usr \
>> --enable-static \
>> 2>&1 configure.log
>>
>> Note, "--enable-static" means OpenMPI will build static libraries
>> (besides the shared ones).
>> OpenMPI is not being linked statically to system libraries,
>> or to IB libraries, etc.
>>
>> Some switches may not be needed,
>> in particularly the explicit use of /usr directory.
>> However, at some point the OpenMPI configure
>> would not work without being
>> told this (at least for libnuma).
>>
>> BTW, I didn't claim your OpenMPI doesn't have IB support.
>> Not a categorical syllogism like
>> "you don't have the -libverbs flag, hence you don't have IB".
>> It is hard to make definitive statements like this
>> in a complex environment like this (OpenMPI build, parallel programs),
>> and with limited information via email.
>> After all, the list is peer reviewed! :)
>> Hence, I only guessed, as I usually do in these exchanges.
>> However, considering all the trouble you've been through, who knows,
>> maybe it was a guess in the right direction.
>>
>> I wonder if there may still be a glitch in the OpenMPI configure
>> script, on how it searches for and uses libraries like IB, NUMA, etc,
>> which may be causing the problem.
>> Jeff: Is this possible?
>>
>> In any case, we have different "Wrapper extra LIBS".
>> I have -lrdmacm -libverbs, you and Noam don't have them.
>> (Noam: I am not saying you don't have IB support! :))
>> My configure explicitly asks for ib support, Noam's (and maybe yours)
>> doesn't.
>> Somehow, slight differences in how one invokes
>> the configure script seems to produce different results.
>>
>> I hope this helps,
>> Gus Correa
>> ---------------------------------------------------------------------
>> Gustavo Correa
>> Lamont-Doherty Earth Observatory - Columbia University
>> Palisades, NY, 10964-8000 - USA
>> ---------------------------------------------------------------------
>>>> -----Original Message-----
>>>> From: users-bounces_at_[hidden]
>>>> [mailto:users-bounces_at_[hidden]] On Behalf Of Noam Bernstein
>>>> Sent: Wednesday, June 24, 2009 9:38 AM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] 50% performance reduction due to
>>>> OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead
>>>> of using Infiniband
>>>>
>>>>
>>>> On Jun 23, 2009, at 6:19 PM, Gus Correa wrote:
>>>>
>>>>> Hi Jim, list
>>>>>
>>>>> On my OpenMPI 1.3.2 ompi_info -config gives:
>>>>>
>>>>> Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl -Wl,--
>>>>> export-dynamic -lnsl -lutil -lm -ldl
>>>>>
>>>>> Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs
>>>>>
>>>>> So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.
>>>> The second of these statements doesn't follow from the first.
>>>>
>>>> My "ompi_info -config" returns
>>>>
>>>> ompi_info -config | grep LIBS
>>>> Build LIBS: -lnsl -lutil -lm
>>>> Wrapper extra LIBS: -ldl -Wl,--export-dynamic
>>>> -lnsl -lutil -
>>>> lm -ldl
>>>>
>>>> But it does have openib
>>>>
>>>> ompi_info | grep openib
>>>> MCA btl: openib (MCA v2.0, API v2.0,
>>>> Component v1.3.2)
>>>>
>>>> and osu_bibw returns
>>>>
>>>> # OSU MPI Bi-Directional Bandwidth Test v3.0
>>>> # Size Bi-Bandwidth (MB/s)
>>>> 4194304 1717.43
>>>>
>>>> which it's sure not getting over ethernet. I think Jeff
>>>> Squyres' test (ompi_info | grep openib) must be more definitive.
>>>>
>>>>
>>>> Noam
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users