Subject: Re: [OMPI users] Openmpi and processor affinity
From: Ashley Pittman (ashley_at_[hidden])
Date: 2009-06-03 11:40:27
On Wed, 2009-06-03 at 11:27 -0400, Jeff Squyres wrote:
> On Jun 3, 2009, at 10:48 AM, <JACOB_LIBERMAN_at_[hidden]> wrote:
>
> > For HPL, try writing a bash script that pins processes to their
> > local memory controllers using numactl before kicking off HPL. This
> > is particularly helpful when spawning more than 1 thread per
> > process. The last line of your script should look like "numactl -c
> > $cpu_bind -m $ mem_bind $*".
> >
> > Believe it or not, I hit 94.5% HPL efficiency using this tactic on a
> > 16 node cluster. Using processor affinity (various MPIs) my results
> > were inconsistent and ranged between 88-93%
> >
>
> If you're using multi-threaded HPL, that might be useful. But if
> you're not, I'd be surprised if you got any different results than
> Open MPI binding itself. If there really is a difference, we should
> figure out why. More specifically, calling numactl yourself should be
> pretty much exactly what we do in OMPI (via API, not via calling
> numactl).
Wasn't there a discussion about this recently on the list, OMPI binds
during MPI_Init() so it's possible for memory to be allocated on the
wrong quad, the discussion was about moving the binding to the orte
process as I recall?
>From my testing of process affinity you tend to get much more consistent
results with it on and much more unpredictable results with it off, I'd
questing that it's working properly if you are seeing a 88-93% range in
the results.
Ashley Pittman.