|
« Return to documentation listing
Table of Contents
ORTE_HOSTS - OpenRTE Hostfile and HOST Behavior: Overview
of OpenRTE’s support for user-supplied hostfiles and comma-delimited lists
of hosts
OpenRTE supports several levels of user-specified
host lists based on an established precedence order. Users can specify a
default hostfile that contains a list of nodes available to all app_contexts
given on the command line. Only one default hostfile can be provided for
any job. In addition, users can specify a hostfile that contains a list
of nodes to be used for a specific app_context, or can provide a comma-delimited
list of nodes to be used for that app_context via the -host command line
option.
The precedence order applied to these various options depends to
some extent on the local environment. The following table illustrates how
host and hostfile directives work together to define the set of hosts upon
which a job will execute in the absence of a resource manager (RM):
default
hostfile host hostfile Result
---------- ------ ---------- -----------------------------------------
unset unset unset Job is co-located with mpirun
unset set unset Host defines resource list for
the job
unset unset set Hostfile defines resource list
for the job
unset set set Hostfile defines resource list
for the job,
then host filters the list to
define the final
set of nodes available to each
application
within the job
set unset unset Default hostfile defines resource
list for the job
set set unset Default hostfile defines resource
list for the job,
then host filters the list to
define the final
set of nodes available to each
application
within the job
set set set Default hostfile defines resource
list for the job,
then hostfile filters the list,
and then host filters
the list to define the final
set of nodes available
to each application within the
job
This changes somewhat in the presence of a RM as that entity specifies
the initial allocation of nodes. In this case, the default hostfile, hostfile
and host directives are all used to filter the RM’s specification so that
a user can utilize different portions of the allocation for different jobs.
This is done according to the same precedence order as in the prior table,
with the RM providing the initial pool of nodes.
Once
an initial allocation has been specified (whether by an RM, default hostfile,
or hostfile), subsequent hostfile and -host specifications can be made using
relative indexing. This allows a user to stipulate which hosts are to be
used for a given app_context without specifying the particular host name,
but rather its relative position in the allocation.
This can probably best
be understood through consideration of a few examples. Consider the case
where an RM has allocated a set of nodes to the user named "foo1, foo2,
foo3, foo4". The user wants the first app_context to have exclusive use
of the first two nodes, and a second app_context to use the last two nodes.
Of course, the user could printout the allocation to find the names of
the nodes allocated to them and then use -host to specify this layout, but
this is cumbersome and would require hand-manipulation for every invocation.
A simpler method is to utilize OpenRTE’s relative indexing capability to
specify the desired layout. In this case, a command line of:
mpirun -pernode
-host +n1,+n2 ./app1 : -host +n3,+n4 ./app2
would provide the desired pattern.
The "+" syntax indicates that the information is being provided as a relative
index to the existing allocation. Two methods of relative indexing are supported:
- +n<#>
- A relative index into the allocation referencing the <#> node. OpenRTE
will substitute the <#> node in the allocation
- +e[:<#>]
- A request for <#> empty
nodes - i.e., OpenRTE is to substitute this reference with <#> nodes that have
not yet been used by any other app_context. If the ":<#>" is not provided,
OpenRTE will substitute the reference with all empty nodes. Note that OpenRTE
does track the empty nodes that have been assigned in this manner, so multiple
uses of this option will result in assignment of unique nodes up to the
limit of the available empty nodes. Requests for more empty nodes than are
available will generate an error.
Relative indexing can be combined with
absolute naming of hosts in any arbitrary manner, and can be used in hostfiles
as well as with the -host command line option. In addition, any slot specification
provided in hostfiles will be respected - thus, a user can specify that
only a certain number of slots from a relative indexed host are to be used
for a given app_context.
Another example may help illustrate this point.
Consider the case where a user has a default hostfile containing:
dummy1 slots=4
dummy2 slots=4
dummy3 slots=4
dummy4 slots=4
dummy5 slots=4
This may, for example, be a hostfile that describes a set of commonly-used
resources that the user wishes to execute applications against. For this
particular application, the user plans to map byslot, and wants the first
two ranks to be on the second node of any allocation, the next ranks to
land on an empty node, have one rank specifically on dummy4, the next rank
to be on the second node of the allocation again, and finally any remaining
ranks to be on whatever empty nodes are left. To accomplish this, the user
provides a hostfile of:
+n2 slots=2
+e:1
dummy4 slots=1
+n2
+e
The user can now use this information in combination with OpenRTE’s sequential
mapper to obtain their specific layout:
mpirun --default-hostfile dummyhosts -hostfile mylayout -mca rmaps seq ./my_app
which will result in:
rank0 being mapped to dummy3
rank1 to dummy1 as the first empty node
rank2 to dummy4
rank3 to dummy3
rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes
Note that the sequential mapper ignores the number of slots arguments as
it only maps one rank at a time to each node in the list.
If the default
round-robin mapper had been used, then the mapping would have resulted in:
ranks 0 and 1 being mapped to dummy3 since two slots were specified
ranks 2-5 on dummy1 as the first empty node, which has four slots
rank6 on dummy4 since the hostfile specifies only a single slot from that
node is to be used
ranks 7 and 8 on dummy3 since only two slots remain available
ranks 9-12 on dummy2 since it is the next available empty node and has four
slots
ranks 13-16 on dummy5 since it is the last remaining unused node and has
four slots
Thus, the use of relative indexing can allow for complex mappings to be
ported across allocations, including those obtained from automated resource
managers, without the need for manual manipulation of scripts and/or command
lines.
orterun(1)
Table of Contents
« Return to documentation listing
|
