Subject: Re: [OMPI users] 50% performance reduction due to OpenMPI v 1.3.2 forcing all MPI traffic over Ethernet instead of using Infiniband
From: Gus Correa (gus_at_[hidden])
Date: 2009-06-23 18:19:47
Hi Jim, list
On my OpenMPI 1.3.2 ompi_info -config gives:
Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl
-Wl,--export-dynamic -lnsl -lutil -lm -ldl
Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs
So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.
That is that is the point I and Scott Hamilton (in the Rocks list)
have been pointing out to you as a possible reason for your problem.
FYI, the OpenMPI configure was changed somewhere between 1.2.8
and 1.3.2 in the way it searches for libraries.
Your configure command line for 1.2.8 may not work as you expect
in 1.3.2.
If your IB libraries are in a non-standard location, you may
need to point to it explicitly.
See the README file and the FAQs.
Just to make sure, I suggest you dump the OpenMPI configure output
to a file, then search it for the IB libraries.
This helped me troubleshoot similar problems (with libnuma) here.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
Jim Kress ORG wrote:
> For v 1.3.2:
>
> Here is the ompi_info -config output and I've attached a copy of the
> config.log file which seems to clearly indicate it found the infiniband
> libraries.
>
> [root_at_master ~]# ompi_info -config
> Configured by: root
> Configured on: Sun Jun 21 22:02:59 EDT 2009
> Configure host: master.org
> Built by: root
> Built on: Sun Jun 21 22:10:07 EDT 2009
> Built host: master.org
> C bindings: yes
> C++ bindings: yes
> Fortran77 bindings: yes (all)
> Fortran90 bindings: yes
> Fortran90 bindings size: small
> C compiler: gcc
> C compiler absolute: /usr/bin/gcc
> C char size: 1
> C bool size: 1
> C short size: 2
> C int size: 4
> C long size: 8
> C float size: 4
> C double size: 8
> C pointer size: 8
> C char align: 1
> C bool align: 1
> C int align: 4
> C float align: 4
> C double align: 8
> C++ compiler: g++
> C++ compiler absolute: /usr/bin/g++
> Fortran77 compiler: gfortran
> Fortran77 compiler abs: /usr/bin/gfortran
> Fortran90 compiler: gfortran
> Fortran90 compiler abs: /usr/bin/gfortran
> Fort integer size: 4
> Fort logical size: 4
> Fort logical value true: 1
> Fort have integer1: yes
> Fort have integer2: yes
> Fort have integer4: yes
> Fort have integer8: yes
> Fort have integer16: no
> Fort have real4: yes
> Fort have real8: yes
> Fort have real16: no
> Fort have complex8: yes
> Fort have complex16: yes
> Fort have complex32: no
> Fort integer1 size: 1
> Fort integer2 size: 2
> Fort integer4 size: 4
> Fort integer8 size: 8
> Fort integer16 size: -1
> Fort real size: 4
> Fort real4 size: 4
> Fort real8 size: 8
> Fort real16 size: -1
> Fort dbl prec size: 4
> Fort cplx size: 4
> Fort dbl cplx size: 4
> Fort cplx8 size: 8
> Fort cplx16 size: 16
> Fort cplx32 size: -1
> Fort integer align: 4
> Fort integer1 align: 1
> Fort integer2 align: 2
> Fort integer4 align: 4
> Fort integer8 align: 8
> Fort integer16 align: -1
> Fort real align: 4
> Fort real4 align: 4
> Fort real8 align: 8
> Fort real16 align: -1
> Fort dbl prec align: 4
> Fort cplx align: 4
> Fort dbl cplx align: 4
> Fort cplx8 align: 4
> Fort cplx16 align: 8
> Fort cplx32 align: -1
> C profiling: yes
> C++ profiling: yes
> Fortran77 profiling: yes
> Fortran90 profiling: yes
> C++ exceptions: no
> Thread support: posix (mpi: no, progress: no)
> Sparse Groups: no
> Build CFLAGS: -O3 -DNDEBUG -finline-functions
> -fno-strict-aliasing
> -pthread -fvisibility=hidden
> Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
> Build FFLAGS:
> Build FCFLAGS:
> Build LDFLAGS: -export-dynamic
> Build LIBS: -lnsl -lutil -lm
> Wrapper extra CFLAGS: -pthread
> Wrapper extra CXXFLAGS: -pthread
> Wrapper extra FFLAGS: -pthread
> Wrapper extra FCFLAGS: -pthread
> Wrapper extra LDFLAGS:
> Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
> -ldl
> Internal debug support: no
> MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
> libltdl support: yes
> Heterogeneous support: no
> mpirun default --prefix: no
> MPI I/O support: yes
> MPI_WTIME support: gettimeofday
> Symbol visibility support: yes
> FT Checkpoint support: no (checkpoint thread: no)
> [root_at_master ~]#
>
>
>
> On Tue, 2009-06-23 at 15:20 -0600, Ralph Castain wrote:
>> Hmmm...just to be clear - did you run this against OMPI 1.3.2, or
>> 1.2.8? I see a 1.2.8 in your app name, hence the question.
>>
>> This option only works with 1.3.2, I'm afraid - it was a new feature.
>>
>> Ralph
>>
>> On Jun 23, 2009, at 2:31 PM, Jim Kress ORG wrote:
>>
>>> Ralph,
>>>
>>> I did the following:
>>>
>>> export OMPI_MCA_mpi_show_mca_params="file,env"
>>>
>>> then I checked and found it via the set command as
>>>
>>> OMPI_MCA_mpi_show_mca_params=file,env
>>>
>>> I then ran my application
>>>
>>> ./orca hexatriene_TDDFT_get_asa_input_parallel_1.inp >
>>> 1.2.8_test_crafted_input_file.out
>>>
>>> and got the expected ORCA output in the .out file but nothing at the
>>> command line or in the .out file about mca_params
>>>
>>> What did I do wrong?
>>>
>>> Jim
>>>
>>> On Mon, 2009-06-22 at 19:40 -0600, Ralph Castain wrote:
>>>> Sounds very strange, indeed. You might want to check that your app is
>>>> actually getting the MCA param that you think it is. Try adding:
>>>>
>>>> -mca mpi_show_mca_params file,env
>>>>
>>>> to your cmd line. This will cause rank=0 to output the MCA params it
>>>> thinks were set via the default files and/or environment (including
>>>> cmd line).
>>>>
>>>> Ralph
>>>>
>>>> On Jun 22, 2009, at 6:14 PM, Jim Kress ORG wrote:
>>>>
>>>>> For the app I am using, ORCA (a Quantum Chemistry program), when it
>>>>> was
>>>>> compiled using openMPI 1.2.8 and run under 1.2.8 with the
>>>>> following in
>>>>> the openmpi-mca-params.conf file:
>>>>>
>>>>> btl=self,openib
>>>>>
>>>>> the app ran fine with no traffic over my Ethernet network and all
>>>>> traffic over my Infiniband network.
>>>>>
>>>>> However, now that ORCA has been recompiled with openMPI v1.3.2 and
>>>>> run
>>>>> under 1.3.2 (using the same openmpi-mca-params.conf file), the
>>>>> performance has been reduced by 50% and all the MPI traffic is going
>>>>> over the Ethernet network.
>>>>>
>>>>> As a matter of fact, the openMPI v1.3.2 performance now looks
>>>>> exactly
>>>>> like the performance I get if I use MPICH 1.2.7.
>>>>>
>>>>> Anyone have any ideas:
>>>>>
>>>>> 1) How could this have happened?
>>>>>
>>>>> 2) How can I fix it?
>>>>>
>>>>> a 50% reduction in performance is just not acceptable. Ideas/
>>>>> suggestions would be appreciated.
>>>>>
>>>>> Jim
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users