Subject: Re: [OMPI users] How to use Multiple links withOpenMPI??????????????????
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-06-01 21:18:14
On May 29, 2009, at 12:31 AM, shan axida wrote:
> Is it true to use bidirectianal communication with MPI in ethernet
> Cluster?
Are you asking if Open MPI uses bi-direction TCP sockets? Yes, it
does: we open one TCP socket between the MPI sender and receiver, and
if the order is reversed (receiver becomes sender), we'll use the same
socket.
> I have tried once (I thought, it is possible because of fully duplex
> swithes).
> However, I could not get bandwidth improvement as I was expecting.
If you really are using hubs, then if you have processes A and B both
sending to each other simultaneously across the same link, you're
going to have contention and one of them will have to wait.
Even if you do have switches, there is a *wide* performance variation
of low-quality switches. Most low-cost ethernet 1GB switches perform
correctly, but do not necessarily provide the same high performance
that you can get with higher-cost switches (i.e., you get what you pay
for).
> If you answer is YES, would you please tell me about pseudocode for
> bidirectional communication ?
>
> Thank you.
> Axida
>
>
>
> From: Jeff Squyres <jsquyres_at_[hidden]>
> To: Open MPI Users <users_at_[hidden]>
> Sent: Wednesday, May 27, 2009 11:28:42 PM
> Subject: Re: [OMPI users] How to use Multiple links with
> OpenMPI??????????????????
>
> Open MPI considers hosts differently than network links.
>
> So you should only list the actual hostname in the hostfile, with
> slots equal to the number of processors (4 in your case, I think?).
>
> Once the MPI processes are launched, they each look around on the
> host that they're running and find network paths to each of their
> peers. If they are multiple paths between pairs of peers, Open MPI
> will round-robin stripe messages across each of the links. We don't
> really have an easy setting for each peer pair only using 1 link.
> Indeed, since connectivity is bidirectional, the traffic patterns
> become less obvious if you want MPI_COMM_WORLD rank X to only use
> link Y -- what does that mean to the other 4 MPI processes on the
> other host (with whom you have assumedly assigned their own
> individual links as well)?
>
>
> On May 26, 2009, at 12:24 AM, shan axida wrote:
>
> > Hi everyone,
> > I want to ask how to use multiple links (multiple NICs) with
> OpenMPI.
> > For example, how can I assign a link to each process, if there are
> 4 links
> > and 4 processors on each node in our cluster?
> > Is this a correct way?
> > hostfile:
> > ----------------------
> > host1-eth0 slots=1
> > host1-eth1 slots=1
> > host1-eth2 slots=1
> > host1-eth3 slots=1
> > host2-eth0 slots=1
> > host2-eth1 slots=1
> > host2-eth2 slots=1
> > host2-eth3 slots=1
> > ... ...
> > ... ...
> > host16-eth0 slots=1
> > host16-eth1 slots=1
> > host16-eth2 slots=1
> > host16-eth3 slots=1
> > ------------------------
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
-- Jeff Squyres Cisco Systems