Chap_API_Key_Value_Mgmt.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Chapter: Key/Value Management
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Key/Value Management}
\label{chap:api_kv_mgmt}

Management of key-value pairs in \ac{PMIx} is a distributed responsibility. While the stated objective of the \ac{PMIx} community is to eliminate collective operations, it is recognized that the traditional method of publishing/exchanging data must be supported until that objective can be met. This method relies on processes to discover and publish their local information which is collected by the local PMIx server library.
Global exchange of the published information is then executed via a collective operation performed by the host \ac{SMS} servers.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Setting and Accessing Key/Value Pairs}
\label{chap:api_kv_mgmt:access}


%%%%%%%%%%%
\subsection{\code{PMIx_Put}}
\declareapi{PMIx_Put}

%%%%
\summary

Push a key/value pair into the client's namespace.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Put(pmix_scope_t scope,
         const pmix_key_t key,
         pmix_value_t *val)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{scope}{Distribution scope of the provided value (handle)}
\argin{key}{key (\refstruct{pmix_key_t})}
\argin{value}{Reference to a \refstruct{pmix_value_t} structure (handle)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

%%%%
\descr

Push a value into the client's namespace.
The client's \ac{PMIx} library will cache the information locally until \refapi{PMIx_Commit} is called.

The provided \refarg{scope} is passed to the local PMIx server, which will distribute the data to other processes according to the provided scope.
The \refstruct{pmix_scope_t} values are defined in \specrefstruct{pmix_scope_t}.
Specific implementations may support different scope values, but all implementations must support at least \code{PMIX_GLOBAL}.

The \refstruct{pmix_value_t} structure supports both string and binary values.
PMIx implementations will support heterogeneous environments by properly converting binary values between host architectures, and will copy the provided \refarg{value} into internal memory.

\adviceimplstart
The PMIx server library will properly pack/unpack data to accommodate heterogeneous environments. The host \ac{SMS} is not involved in this action. The \refarg{value} argument must be copied - the caller is free to release it following return from the function.
\adviceimplend

\adviceuserstart
The value is copied by the PMIx client library. Thus, the application is free to release and/or modify the value once the call to \refapi{PMIx_Put} has completed.

Note that keys starting with a string of ``\code{pmix}'' are exclusively reserved for the \ac{PMIx} standard and must not be used in calls to \refapi{PMIx_Put}. Thus, applications should never use a defined ``PMIX_'' attribute as the key in a call to \refapi{PMIx_Put}.
\adviceuserend


%%%%%%%%%%%
\subsection{\code{PMIx_Get}}
\declareapi{PMIx_Get}

%%%%
\summary

Retrieve a key/value pair from the client's namespace.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Get(const pmix_proc_t *proc, const pmix_key_t key,
         const pmix_info_t info[], size_t ninfo,
         pmix_value_t **val)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{proc}{process reference (handle)}
\argin{key}{key to retrieve (\refstruct{pmix_key_t})}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\argout{val}{value (handle)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pastePRIAttributeItem{PMIX_OPTIONAL}
\pastePRIAttributeItem{PMIX_IMMEDIATE}
\pastePRIAttributeItem{PMIX_DATA_SCOPE}

\reqattrend

\optattrstart
The following attributes are optional for host environments:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between delivery of the data by the host environment versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Retrieve information for the specified \refarg{key} as published by the process identified in the given \refstruct{pmix_proc_t}, returning a pointer to the value in the given address.

This is a blocking operation - the caller will block until either the specified data becomes available from the specified rank in the \refarg{proc} structure or the operation times out should the \refattr{PMIX_TIMEOUT} attribute have been given.
The caller is responsible for freeing all memory associated with the returned \refarg{value} when no longer required.

The \refarg{info} array is used to pass user requests regarding the get operation.

\adviceuserstart
Information provided by the \ac{PMIx} server at time of process start is accessed by providing the namespace of the job with the rank set to \refconst{PMIX_RANK_WILDCARD}. The list of data referenced in this way is maintained on the \ac{PMIx} web site at \url{https://pmix.org/support/faq/wildcard-rank-access/} but includes items such as the number of processes in the namespace (\refattr{PMIX_JOB_SIZE}), total available slots in the allocation (\refattr{PMIX_UNIV_SIZE}), and the number of nodes in the allocation (\refattr{PMIX_NUM_NODES}).

In general, only data posted by a process via \refapi{PMIx_Put} needs to be retrieved by specifying the rank of the posting process. All other information is retrievable using a rank of \refconst{PMIX_RANK_WILDCARD}.
\adviceuserend

%%%%%%%%%%%
\subsection{\code{PMIx_Get_nb}}
\declareapi{PMIx_Get_nb}

%%%%
\summary

Nonblocking \refapi{PMIx_Get} operation.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Get_nb(const pmix_proc_t *proc, const char key[],
            const pmix_info_t info[], size_t ninfo,
            pmix_value_cbfunc_t cbfunc, void *cbdata)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{proc}{process reference (handle)}
\argin{key}{key to retrieve (string)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pastePRIAttributeItem{PMIX_OPTIONAL}
\pastePRIAttributeItem{PMIX_IMMEDIATE}
\pastePRIAttributeItem{PMIX_DATA_SCOPE}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between delivery of the data by the host environment versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

The callback function will be executed once the specified data becomes available from the identified process and retrieved by the local server.
The \argref{info} array is used as described by the \refapi{PMIx_Get} routine.

\adviceuserstart
Information provided by the \ac{PMIx} server at time of process start is accessed by providing the namespace of the job with the rank set to \refconst{PMIX_RANK_WILDCARD}. The list of data referenced in this way is maintained on the \ac{PMIx} web site at \url{https://pmix.org/support/faq/wildcard-rank-access/} but includes items such as the number of processes in the namespace (\refattr{PMIX_JOB_SIZE}), total available slots in the allocation (\refattr{PMIX_UNIV_SIZE}), and the number of nodes in the allocation (\refattr{PMIX_NUM_NODES}).

In general, only data posted by a process via \refapi{PMIx_Put} needs to be retrieved by specifying the rank of the posting process. All other information is retrievable using a rank of \refconst{PMIX_RANK_WILDCARD}.
\adviceuserend


%%%%%%%%%%%
\subsection{\code{PMIx_Store_internal}}
\declareapi{PMIx_Store_internal}

%%%%
\summary

Store some data locally for retrieval by other areas of the proc.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Store_internal(const pmix_proc_t *proc,
                    const pmix_key_t key,
                    pmix_value_t *val);
\end{codepar}
\cspecificend

\begin{arglist}
\argin{proc}{process reference (handle)}
\argin{key}{key to retrieve (string)}
\argin{val}{Value to store (handle)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

%%%%
\descr

Store some data locally for retrieval by other areas of the proc.
This is data that has only internal scope - it will never be ``pushed'' externally.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Exchanging Key/Value Pairs}
\label{chap:api_kv_mgmt:exchange}

The APIs defined in this section push key/value pairs from the client to the local \ac{PMIx} server, and circulate the data between \ac{PMIx} servers for subsequent retrieval by the local clients.

%%%%%%%%%%%
\subsection{\code{PMIx_Commit}}
\declareapi{PMIx_Commit}

%%%%
\summary

Push all previously \refapi{PMIx_Put} values to the local PMIx server.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t PMIx_Commit(void)
\end{codepar}
\cspecificend

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

%%%%
\descr

This is an asynchronous operation.
The \ac{PRI} will immediately return to the caller while the data is transmitted to the local server in the background.

\adviceuserstart
The local PMIx server will cache the information locally - i.e., the committed data will not be circulated during \refapi{PMIx_Commit}.
Availability of the data upon completion of \refapi{PMIx_Commit} is therefore implementation-dependent.
\adviceuserend


%%%%%%%%%%%
\subsection{\code{PMIx_Fence}}
\declareapi{PMIx_Fence}

%%%%
\summary

Execute a blocking barrier across the processes identified in the specified array, collecting information posted via \refapi{PMIx_Put} as directed.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Fence(const pmix_proc_t procs[], size_t nprocs,
           const pmix_info_t info[], size_t ninfo)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)}
\argin{nprocs}{Number of element in the \refarg{procs} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pastePRIAttributeItem{PMIX_COLLECT_DATA}

\reqattrend

\optattrstart
The following attributes are optional for host environments:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pasteAttributeItem{PMIX_COLLECTIVE_ALGO}
\pasteAttributeItem{PMIX_COLLECTIVE_ALGO_REQD}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Passing a \code{NULL} pointer as the \refarg{procs} parameter indicates that the fence is to span all processes in the client's namespace.
Each provided \refstruct{pmix_proc_t} struct can pass \refconst{PMIX_RANK_WILDCARD} to indicate that all processes in the given namespace are participating.

The \refarg{info} array is used to pass user requests regarding the fence operation.

Note that for scalability reasons, the default behavior for \refapi{PMIx_Fence} is to \emph{not} collect the data.


%%%%%%%%%%%
\subsection{\code{PMIx_Fence_nb}}
\declareapi{PMIx_Fence_nb}

%%%%
\summary

Execute a nonblocking \refapi{PMIx_Fence} across the processes identified in the specified array of processes, collecting information posted via \refapi{PMIx_Put} as directed.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Fence_nb(const pmix_proc_t procs[], size_t nprocs,
              const pmix_info_t info[], size_t ninfo,
              pmix_op_cbfunc_t cbfunc, void *cbdata)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{procs}{Array of \refstruct{pmix_proc_t} structures (array of handles)}
\argin{nprocs}{Number of element in the \refarg{procs} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
The following attributes are required to be supported by all \ac{PMIx} libraries:

\pastePRIAttributeItem{PMIX_COLLECT_DATA}

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pasteAttributeItem{PMIX_COLLECTIVE_ALGO}
\pasteAttributeItem{PMIX_COLLECTIVE_ALGO_REQD}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Nonblocking \refapi{PMIx_Fence} routine.
Note that the function will return an error if a \code{NULL} callback function is given.

Note that for scalability reasons, the default behavior for \refapi{PMIx_Fence_nb} is to \emph{not} collect the data.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Publish and Lookup Data}
\label{chap:api_kv_mgmt:publish}

The APIs defined in this section publish data from one client that can be later exchanged and looked up by another client.

\adviceimplstart
\ac{PMIx} libraries that support any of the functions in this section are required to support \textit{all} of them.
\adviceimplend

\advicermstart
Host environments that support any of the functions in this section are required to support \textit{all} of them.
\advicermend

%%%%%%%%%%%
\subsection{\code{PMIx_Publish}}
\declareapi{PMIx_Publish}

%%%%
\summary

Publish data for later access via \refapi{PMIx_Lookup}.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Publish(const pmix_info_t info[], size_t ninfo)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that published the info.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}
\pastePRRTEAttributeItem{PMIX_PERSISTENCE}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Publish the data in the \refarg{info} array for subsequent lookup.
By default, the data will be published into the \refconst{PMIX_SESSION} range and with \refconst{PMIX_PERSIST_APP} persistence.
Changes to those values, and any additional directives, can be included in the \refstruct{pmix_info_t} array. Attempts to access the data by processes outside of the provided data range will be rejected. The persistence parameter instructs the server as to how long the data is to be retained.

The blocking form will block until the server confirms that the data has been sent to the \ac{PMIx} server and that it has obtained confirmation from its host \ac{SMS} daemon that the data is ready to be looked up. Data is copied into the backing key-value data store, and therefore the \refarg{info} array can be released upon return from the blocking function call.

\adviceuserstart
Duplicate keys within the specified data range may lead to unexpected behavior depending
on host RM implementation of the backing key-value store.
\adviceuserend

\adviceimplstart
Implementations should, to the best of their ability, detect duplicate keys and protect the
user from unexpected behavior - preferably returning an error. This version of the
standard does not define a specific error code to be returned, so the implementation must
make it clear to the user what to expect in this scenario. One suggestion is to define an RM
specific error code beyond the \refconst{PMIX_EXTERNAL_ERR_BASE} boundary. Future versions of the standard
will clarify that a specific \ac{PMIx} error be returned when conflicting values are published for a given
key, and will provide attributes to allow modified behaviors such as overwrite.
\adviceimplend

%%%%%%%%%%%
\subsection{\code{PMIx_Publish_nb}}
\declareapi{PMIx_Publish_nb}

%%%%
\summary

Nonblocking \refapi{PMIx_Publish} routine.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Publish_nb(const pmix_info_t info[], size_t ninfo,
                pmix_op_cbfunc_t cbfunc, void *cbdata)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that published the info.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}
\pastePRRTEAttributeItem{PMIX_PERSISTENCE}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Nonblocking \refapi{PMIx_Publish} routine. The non-blocking form will return immediately, executing the callback when the \ac{PMIx} server receives confirmation from its host \ac{SMS} daemon.

Note that the function will return an error if a \code{NULL} callback function is given, and that the \refarg{info} array must be maintained until the callback is provided.


%%%%%%%%%%%
\subsection{\code{PMIx_Lookup}}
\declareapi{PMIx_Lookup}

%%%%
\summary

Lookup information published by this or another process with \refapi{PMIx_Publish} or \refapi{PMIx_Publish_nb}.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Lookup(pmix_pdata_t data[], size_t ndata,
            const pmix_info_t info[], size_t ninfo)
\end{codepar}
\cspecificend

\begin{arglist}
\arginout{data}{Array of publishable data structures (array of handles)}
\argin{ndata}{Number of elements in the \refarg{data} array (integer)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of elements in the \refarg{info} array (integer)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that is requesting the info.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}
\pastePRRTEAttributeItem{PMIX_WAIT}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Lookup information published by this or another process.
By default, the search will be conducted across the \refconst{PMIX_SESSION} range.
Changes to the range, and any additional directives, can be provided in the \refstruct{pmix_info_t} array.

Note that the search is also constrained to only data published by the current user (i.e., the search will not return data published by an application being executed by another user).
There currently is no option to override this behavior - such an option may become available later via an appropriate \refstruct{pmix_info_t} directive.

The \argref{data} parameter consists of an array of \refstruct{pmix_pdata_t} struct with the keys specifying the requested information.
Data will be returned for each key in the associated \refarg{value} struct.
Any key that cannot be found will return with a data type of \refconst{PMIX_UNDEF}.
The function will return \refconst{PMIX_SUCCESS} if \emph{any} values can be found, so the caller must check each data element to ensure it was returned.

The proc field in each \refstruct{pmix_pdata_t} struct will contain the namespace/rank of the process that published the data.

\adviceuserstart
Although this is a blocking function, it will \emph{not} wait by default for the requested data to be published.
Instead, it will block for the time required by the server to lookup its current data and return any found items.
Thus, the caller is responsible for ensuring that data is published prior to executing a lookup, using \refattr{PMIX_WAIT} to instruct the server to wait for the data to be published, or for retrying until the requested data is found.
\adviceuserend

%%%%%%%%%%%
\subsection{\code{PMIx_Lookup_nb}}
\declareapi{PMIx_Lookup_nb}

%%%%
\summary

Nonblocking version of \refapi{PMIx_Lookup}.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Lookup_nb(char **keys,
               const pmix_info_t info[], size_t ninfo,
               pmix_lookup_cbfunc_t cbfunc, void *cbdata)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{keys}{Array to be provided to the callback (array of strings)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function (handle)}
\argin{cbdata}{Callback data to be provided to the callback function (pointer)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that is requesting the info.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}
\pastePRRTEAttributeItem{PMIX_WAIT}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend


%%%%
\descr

Non-blocking form of the \refapi{PMIx_Lookup} function.
Data for the provided NULL-terminated \refarg{keys} array will be returned in the provided callback function.
As with \refapi{PMIx_Lookup}, the default behavior is to \emph{not} wait for data to be published.
The \refarg{info} array can be used to modify the behavior as previously described by \refapi{PMIx_Lookup}. Both the \refarg{info} and \refarg{keys} arrays must be maintained until the callback is provided.



%%%%%%%%%%%
\subsection{\code{PMIx_Unpublish}}
\declareapi{PMIx_Unpublish}

%%%%
\summary

Unpublish data posted by this process using the given keys.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Unpublish(char **keys,
               const pmix_info_t info[], size_t ninfo)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that is requesting the operation.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend


%%%%
\descr

Unpublish data posted by this process using the given \refarg{keys}.
The function will block until the data has been removed by the server (i.e., it is safe to publish that key again).
A value of \code{NULL} for the \refarg{keys} parameter instructs the server to remove \emph{all} data published by this process.

By default, the range is assumed to be \refconst{PMIX_SESSION}.
Changes to the range, and any additional directives, can be provided in the \refarg{info} array.


%%%%%%%%%%%
\subsection{\code{PMIx_Unpublish_nb}}
\declareapi{PMIx_Unpublish_nb}

%%%%
\summary

Nonblocking version of \refapi{PMIx_Unpublish}.

%%%%
\format

\versionMarker{1.0}
\cspecificstart
\begin{codepar}
pmix_status_t
PMIx_Unpublish_nb(char **keys,
                  const pmix_info_t info[], size_t ninfo,
                  pmix_op_cbfunc_t cbfunc, void *cbdata)
\end{codepar}
\cspecificend

\begin{arglist}
\argin{keys}{(array of strings)}
\argin{info}{Array of info structures (array of handles)}
\argin{ninfo}{Number of element in the \refarg{info} array (integer)}
\argin{cbfunc}{Callback function \refapi{pmix_op_cbfunc_t} (function reference)}
\argin{cbdata}{Data to be passed to the callback function (memory reference)}
\end{arglist}

Returns \refconst{PMIX_SUCCESS} or a negative value corresponding to a PMIx error constant.

\reqattrstart
\ac{PMIx} libraries are not required to directly support any attributes for this function. However, any provided attributes must be passed to the host \ac{SMS} daemon for processing, and the \ac{PMIx} library is \textit{required} to add the \refPRIAttributeItem{PMIX_USERID} and the \refPRIAttributeItem{PMIX_GRPID} attributes of the client process that is requesting the operation.

\reqattrend

\optattrstart
The following attributes are optional for host environments that support this operation:

\pastePRRTEAttributeItem{PMIX_TIMEOUT}
\pastePRRTEAttributeItem{PMIX_RANGE}

\optattrend

\adviceimplstart
We recommend that implementation of the \refattr{PMIX_TIMEOUT} attribute be left to the host environment due to race condition considerations between completion of the operation versus internal timeout in the \ac{PMIx} server library. Implementers that choose to support \refattr{PMIX_TIMEOUT} directly in the \ac{PMIx} server library must take care to resolve the race condition and should avoid passing \refattr{PMIX_TIMEOUT} to the host environment so that multiple competing timeouts are not created.
\adviceimplend

%%%%
\descr

Non-blocking form of the \refapi{PMIx_Unpublish} function.
The callback function will be executed once the server confirms removal of the specified data. The \refarg{info} array must be maintained until the callback is provided.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%