Subject: Re: [OMPI users] Best way to overlap computation and transfer using MPIover TCP/Ethernet?
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-06-05 07:30:16
On Jun 4, 2009, at 3:53 AM, Lars Andersson wrote:
> In my second test, I simply put a sleep(3) at point 2), and expected
> the MPI_Wait() call at 3) to finish almost instantly, since I assumed
> that the message would have been transferred during the sleep. To my
> disappointment tough, it took more or less the same time to finish the
> MPI_Wait as without any sleep.
>
As you found by googling, and as Bogdan infers, Open MPI doesn't
currently make much progress over TCP-based networks "in the
background." And you're right that putting an MPI_WAIT in a progress
thread would cause that thread to spin heavily, effectively taking
much of your CPU cycles away from you, and possibly even having other
bad effects (e.g., cache thrashing, context switching, etc.).
I'd say that your own workaround here is to intersperse MPI_TEST's
periodically. This will trigger OMPI's pipelined protocol for large
messages, and should allow partial bursts of progress while you're
assumedly off doing useful work. If this is difficult because the
work is being done in library code that you can't change, then perhaps
a pre-spawned "work" through could be used to call MPI_TEST
periodically. That way, it won't steal huge ammounts of CPU cycles
(like MPI_WAIT would). You still might get some cache thrashing,
context switching, etc. -- YMMV.
As for exactly how many / how often you should call MPI_TEST, that is
going to be up to you. It's going to depend on a lot of factors -- how
big the message is, how well synchronized you are with the receiver,
what strategy you use to call MPI_TEST, etc.
Open MPI may someday treat this better to either have a blocking form
of MPI_WAIT (i.e., not spinning, or spinning considerably less) or
have true TCP progress in the background. But if I had to guess, I'd
say that we'll likely do the former before the latter.
-- Jeff Squyres Cisco Systems