From: Ole Holm Nielsen (Ole.H.Nielsen_at_[hidden])
Date: 2007-05-02 09:57:27
Ralph, thanks very much for your continued support:
Ralph Castain wrote:
>> I'd say that this behavior of mpirun under Torque TM should be considered as
>> a bug. Ideally, users should not have to design their scripts differently
>> according to whether the sysadmin decided to configure in TM or not.
>> Also, for interactive tests one doesn't have TM. I think that mpirun just
>> ought to work no matter what...
>>
>
> In your prior notes, you indicated that you had in fact configured TM
> support into OpenMPI. The issue, therefore, was that you were somehow
> getting an error from TM indicating that the tm spawn command was unable to
> launch our daemon on a specified node.
>
> Your comment above, however, talks about the problem of NOT having TM
> configured into OpenMPI, even though you are running on a Torque-based
> system. This is a significantly different scenario - could you please
> clarify?
What I'm saying is that users should be able run the same script in different
environments, they being Torque or non-Torque, without having to change
the arguments to the mpirun command. Maybe they submit batch jobs to
our Linux/Torque cluster, or maybe they run their scripts on their own
non-Torque workstation. The sysadmin may also reserve a set of nodes in the
Linux cluster and log in interactively (without using Torque) for test
purposes, and in this case the very same mpirun executable file will not
use the TM interface.
IMHO, it is highly desirable that the mpirun command is robust when being run
in different ways, i.e., mpirun should accept both -np and -machinefile
under all circumstances (but preferably print a warning message if it chooses
to ignore -machinefile).
> BTW: We run interactive tests under TM all the time - there is no TM
> requirement prohibiting you from this mode of operation. I would guess,
> therefore, that this may be something your sysadmin has imposed.
>
> Given your comment, however, I must ask: did you get an allocation for the
> nodes in your PBS_NODEFILE prior to executing mpirun??
>
> I need to know since your observed errors could easily be explained by an
> attempt to execute on nodes that are not allocated to you. For example, if
> you either used a PBS_NODEFILE from a prior (possibly batch) execution, or
> created one yourself, Torque would refuse to execute on the specified nodes
> since they aren't allocated to you - i.e., the system would refuse to
> execute the specified executable on that node because you don't have
> permission to do so.
No no: The perfectly correct and sane Torque batch job reported previously in
this thread fails when you execute mpirun like this:
/usr/local/openmpi-1.2.1-pgi/bin/mpirun -np 2 -machinefile $PBS_NODEFILE hello_c
[u126.dcsc.fysik.dtu.dk:11981] pls:tm: failed to poll for a spawned proc, return
status = 17002
[u126.dcsc.fysik.dtu.dk:11981] [0,0,0] ORTE_ERROR_LOG: In errno in file
rmgr_urm.c at line 462
[u126.dcsc.fysik.dtu.dk:11981] mpirun: spawn failed with errno=-11
In this case the $PBS_NODEFILE is created for this job instance by Torque.
The workaround is to omit the perfectly sensible (or at least not
incorrect :-) mpirun arguments "-machinefile $PBS_NODEFILE".
> In that case, we could improve the error message, but the system is actually
> doing everything correctly.
Indeed, except that the above error message is totally unintelligible.
There is no conflict in this job between "-np 2" which refers to 2 specific
nodes allocated by Torque, and "-machinefile $PBS_NODEFILE" which refers
to the very same 2 nodes allocated by Torque. It is beyond me why the
redundant but consistent mpirun node information (in the case of being run
under TM control) would cause mpirun to fail as shown above.
Thanks again,
Ole