Subject: Re: [OMPI users] 50% performance reduction due to OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead of using Infiniband
From: Gus Correa (gus_at_[hidden])
Date: 2009-06-24 17:22:56
Hi Jim
Jim Kress wrote:
> Noam, Gus and List,
>
> Did you statically link your openmpi when you built it? If you did (the
> default is NOT to do this) then that could explain the discrepancy.
>
> Jim
No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
no "-static"):
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
>
>> -----Original Message-----
>> From: users-bounces_at_[hidden]
>> [mailto:users-bounces_at_[hidden]] On Behalf Of Noam Bernstein
>> Sent: Wednesday, June 24, 2009 9:38 AM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] 50% performance reduction due to
>> OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead
>> of using Infiniband
>>
>>
>> On Jun 23, 2009, at 6:19 PM, Gus Correa wrote:
>>
>>> Hi Jim, list
>>>
>>> On my OpenMPI 1.3.2 ompi_info -config gives:
>>>
>>> Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl -Wl,--
>>> export-dynamic -lnsl -lutil -lm -ldl
>>>
>>> Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs
>>>
>>> So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.
>> The second of these statements doesn't follow from the first.
>>
>> My "ompi_info -config" returns
>>
>> ompi_info -config | grep LIBS
>> Build LIBS: -lnsl -lutil -lm
>> Wrapper extra LIBS: -ldl -Wl,--export-dynamic
>> -lnsl -lutil -
>> lm -ldl
>>
>> But it does have openib
>>
>> ompi_info | grep openib
>> MCA btl: openib (MCA v2.0, API v2.0,
>> Component v1.3.2)
>>
>> and osu_bibw returns
>>
>> # OSU MPI Bi-Directional Bandwidth Test v3.0
>> # Size Bi-Bandwidth (MB/s)
>> 4194304 1717.43
>>
>> which it's sure not getting over ethernet. I think Jeff
>> Squyres' test (ompi_info | grep openib) must be more definitive.
>>
>>
>> Noam
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users