Subject: Re: [OMPI users] How to use Multiple links withOpenMPI??????????????????
From: shan axida (axida2009_at_[hidden])
Date: 2009-06-04 03:42:49
Hi Jeff Squyres,
We have Dell powerconnect 2724 Gigabit switches to connect the nodes in our cluster.
As you said, may be the speed of PCI bus is a bottleneck.
How can check it in practical?
What is your suggestion for the problem?
Thank you!
Axida
________________________________
From: Jeff Squyres <jsquyres_at_[hidden]>
To: Open MPI Users <users_at_[hidden]>
Sent: Tuesday, June 2, 2009 10:15:39 AM
Subject: Re: [OMPI users] How to use Multiple links withOpenMPI??????????????????
Note that striping doesn't really help you much until data sizes get large. For example, networks tend to have an elbow in the graph where the size of the message starts to matter (clearly evident on your graphs).
Additionally, you have your network marked as with "hubs" not "switches" -- if you really do have hubs and not switches, you may run into serious contention issues if you start loading up the network.
With both of these factors, even though you have 4 links, you likely aren't going to see much of a performance benefit until you send large messages (which will be limited by your bus speeds -- can you feed all 4 of your links from a single machine at line rate, or will you be limited by PCI bus speeds and contention?), and you may run into secondary performance issues due to contention on your hubs.
On May 28, 2009, at 11:06 PM, shan axida wrote:
> Thank you! Mr. Jeff Squyres,
> I have conducted a simple MPI_Bcast experiment in out cluster.
> The results are shown in the file attached on this e-mail.
> The hostfile is :
> -----------------
> hostname1 slots=4
> hostname2 slots=4
> hostname3 slots=4
> ....
> ....
> hostname16 slots=4
> -----------------
> As we can see in the figure, it is little faster than single link
> when we use 2,3,4 links between nodes.
> My question is what would be the reason to make almost the same
> performance when we use 2,3,4 links ?
>
> Thank you!
>
> Axida
>
>
>
>
> From: Jeff Squyres <jsquyres_at_[hidden]>
> To: Open MPI Users <users_at_[hidden]>
> Sent: Wednesday, May 27, 2009 11:28:42 PM
> Subject: Re: [OMPI users] How to use Multiple links with OpenMPI??????????????????
>
> Open MPI considers hosts differently than network links.
>
> So you should only list the actual hostname in the hostfile, with slots equal to the number of processors (4 in your case, I think?).
>
> Once the MPI processes are launched, they each look around on the host that they're running and find network paths to each of their peers. If they are multiple paths between pairs of peers, Open MPI will round-robin stripe messages across each of the links. We don't really have an easy setting for each peer pair only using 1 link. Indeed, since connectivity is bidirectional, the traffic patterns become less obvious if you want MPI_COMM_WORLD rank X to only use link Y -- what does that mean to the other 4 MPI processes on the other host (with whom you have assumedly assigned their own individual links as well)?
>
>
> On May 26, 2009, at 12:24 AM, shan axida wrote:
>
> > Hi everyone,
> > I want to ask how to use multiple links (multiple NICs) with OpenMPI.
> > For example, how can I assign a link to each process, if there are 4 links
> > and 4 processors on each node in our cluster?
> > Is this a correct way?
> > hostfile:
> > ----------------------
> > host1-eth0 slots=1
> > host1-eth1 slots=1
> > host1-eth2 slots=1
> > host1-eth3 slots=1
> > host2-eth0 slots=1
> > host2-eth1 slots=1
> > host2-eth2 slots=1
> > host2-eth3 slots=1
> > ... ...
> > ... ...
> > host16-eth0 slots=1
> > host16-eth1 slots=1
> > host16-eth2 slots=1
> > host16-eth3 slots=1
> > ------------------------
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> <MPI_Bcast-ypc05xx.pdf>_______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--Jeff Squyres
Cisco Systems
_______________________________________________
users mailing list
users_at_[hidden]
http://www.open-mpi.org/mailman/listinfo.cgi/users