Subject: Re: [OMPI users] Openmpi and processor affinity
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-06-03 11:27:25
On Jun 3, 2009, at 10:48 AM, <JACOB_LIBERMAN_at_[hidden]> wrote:
> For HPL, try writing a bash script that pins processes to their
> local memory controllers using numactl before kicking off HPL. This
> is particularly helpful when spawning more than 1 thread per
> process. The last line of your script should look like "numactl -c
> $cpu_bind -m $ mem_bind $*".
>
> Believe it or not, I hit 94.5% HPL efficiency using this tactic on a
> 16 node cluster. Using processor affinity (various MPIs) my results
> were inconsistent and ranged between 88-93%
>
If you're using multi-threaded HPL, that might be useful. But if
you're not, I'd be surprised if you got any different results than
Open MPI binding itself. If there really is a difference, we should
figure out why. More specifically, calling numactl yourself should be
pretty much exactly what we do in OMPI (via API, not via calling
numactl).
-- Jeff Squyres Cisco Systems