ygm::comm

class comm

Public Functions

inline comm(int *argc, char ***argv)

YGM communicator constructor.

#include <ygm/comm.hpp>

int main(int argc, char **argv) {
   ygm::comm world(&argc, &argv);
}

Parameters:

argc – Pointer to number of arguments given to command line
argv – Pointer to array of command line arguments

Returns:

Constructed ygm::comm object using MPI_COMM_WORLD for communication

inline comm(MPI_Comm comm)

YGM communicator constructor.

Parameters:: mcomm – MPI communicator to use for underlying communication
Returns:: Constructed ygm::comm object

inline ~comm()

Destructor for comm object.

Calls a barrier() to ensure all messages have been processed, cancels all outstanding MPI receives and destroys MPI communicators set up for use within the ygm::comm

inline void welcome(std::ostream &os = std::cout)

Prints a welcome message with configuration details.

Prints a YGM welcome statement including information about internal YGM parameters.

Parameters:: os – Output stream to print welcome message to

inline void stats_reset()

Resets counters within the comm_stats object being used by the ygm::comm.

Useful for separating information about communication performed in computation of interest from set-up or from other trials of the same experiment.

inline void stats_print(const std::string &name = "", std::ostream &os = std::cout)

Prints information about communication tracked in comm_stats object.

Parameters:

name – Label to be printed with stats
os – Output stream to print stats to

template<typename AsyncFunction, typename ...SendArgs> inline void async(int dest, AsyncFunction &&fn, const SendArgs&... args)

Asynchronous message initiation.

Serializes function object and queues for sending. Message will be sent and executed at some future time that YGM deems appropriate.

Template Parameters:

AsyncFunction – Type of function object
SendArgs... – Variadic type of arguments to send along with function. All types must be serializable.

Parameters:

dest – Rank to execute function on
fn – Function object to execute at remote destination
args... – Variadic arguments to send with message and pass to function during execution

template<typename AsyncFunction, typename ...SendArgs> inline void async(int dest, AsyncFunction &&fn, const SendArgs&... args) const

template<typename AsyncFunction, typename ...SendArgs> inline void async_bcast(AsyncFunction &&fn, const SendArgs&... args)

Asynchronous message initiation for function that is sent to all ranks.

Serializes function object and queues for sending to all ranks. Message will be sent and executed at some future time that YGM deems appropriate. Messages are sent along an implicitly defined broadcast tree that takes advantage of knowledge of rank assignments to compute nodes.

Template Parameters:

AsyncFunction – Type of function object
SendArgs... – Variadic type of arguments to send along with function. All types must be serializable.

Parameters:

fn – Function object to execute at remote destination
args... – Variadic arguments to send with message and pass to function during execution

template<typename AsyncFunction, typename ...SendArgs> inline void async_bcast(AsyncFunction &&fn, const SendArgs&... args) const

template<typename AsyncFunction, typename ...SendArgs> inline void async_mcast(const std::vector<int> &dests, AsyncFunction &&fn, const SendArgs&... args)

template<typename AsyncFunction, typename ...SendArgs> inline void async_mcast(const std::vector<int> &dests, AsyncFunction &&fn, const SendArgs&... args) const

inline void cf_barrier() const: Control Flow Barrier Only blocks the control flow until all processes in the communicator have called it. See: MPI_Barrier()

inline void barrier()

Full communicator barrier.

Collective operation that processes all messages (including any recursively produced messages) on all ranks. All ranks must complete their messages before any rank is able to return from the barrier() call.

inline void barrier() const: Full communicator barrier that can be called on const comm objects.

inline void async_barrier()

Asynchronous communicator barrier.

An async_barrier can match with other async_barrier and barrier calls. Any comm::barrier() calls matching with any async_barrier will execute as expected but will not return until all ranks are in a non-async barrier. The following code will complete successfully. If the calls to async_barrier() were replaced with barrier(), the code would deadlock with more than 1 rank. This call is useful when ranks may locally decide to run more iterations of a loop than other ranks.

for (int i=0; i<world.size(); ++i) {
  world.async_barrier();
}
world.barrier();

inline void async_barrier() const: Asynchronous communicator barrier that can be called on const comm objects.

inline void local_progress(): Checks for incoming unless called from receive queue and flushes one buffer.

inline bool local_process_incoming()

Check for incoming messages and continue processing until no messages are found.

Returns:: True if any messages were received, otherwise false.

template<typename Function> inline void local_wait_until(Function fn)

Waits until provided condition function returns true.

This is useful when applications can determine locally that their part of a computation is complete (or nearly complete). This can be used to completely avoid barrier() calls or reduce the number of reductions needed within a barrier() to reach quiescence.

   static int messages_received;
   messages_received = 0;
   for (int i=0; i<world.rank(); ++i) {
       world.async(i, [](){++messages_received;});
   }

   world.local_wait_until([&world](){return messages_received ==
world.rank()});

Template Parameters:: Function – functor type
Parameters:: fn – Wait condition function, must match []() -> bool

template<typename T> inline ygm_ptr<T> make_ygm_ptr(T &t)

inline void register_pre_barrier_callback(const std::function<void()> &fn)

Registers a callback that will be executed prior to the barrier completion.

Parameters:: fn – callback function

template<typename T> inline T all_reduce_sum(const T &t) const: Warning

Deprecated

template<typename T> inline T all_reduce_min(const T &t) const: Warning

Deprecated

template<typename T> inline T all_reduce_max(const T &t) const: Warning

Deprecated

template<typename T, typename MergeFunction> inline T all_reduce(const T &t, MergeFunction merge) const: Warning

Deprecated

inline int size() const

Number of ranks in communicator.

Returns:: Communicator size

inline int rank() const

Rank of the current process.

Ranks are unique IDs in the range [0, size-1] assigned to each process in the communicator.

Returns:: Rank within communicator

inline MPI_Comm get_mpi_comm() const

Access to copy of underlying MPI communicator.

Returned MPI_Comm is still managed by YGM and will be freed during ygm::comm destructor.

Returns:: Copy of MPI communicator distinct from one used for asynchronous communication

inline const detail::layout &layout() const

Access to underlying layout object.

Returns:: ygm::detail::layout object used by the ygm::comm

inline const detail::comm_router &router() const

Access to underlying comm_router object.

Returns:: ygm::detail::comm_router object used by the ygm::comm

inline const detail::comm_stats &stats() const

Access to underlying comm_stats object.

Returns:: ygm::detail::comm_stats object used by the ygm::comm

inline bool rank0() const

Checks if current rank is rank 0.

Returns:: bool indicating whether current rank is rank 0

template<typename T> inline void mpi_send(const T &data, int dest, int tag, MPI_Comm comm) const

Send an MPI message.

Template Parameters:

T – datatype being sent (must be serializable with cereal)

Parameters:

data – Message contents to send
dest – Rank to send data to
tag – MPI tag to assign to message
comm – MPI communicator to send message over

template<typename T> inline void mpi_send(const T &data, int dest, int tag) const

Send an MPI message over an unspecified MPI communicator.

Template Parameters:

T – datatype being sent (must be serializable with cereal)

Parameters:

data – Message contents to send
dest – Rank to send data to
tag – MPI tag to assign to message

template<typename T> inline T mpi_recv(int source, int tag, MPI_Comm comm) const

Receive an MPI message.

Template Parameters:

T – datatype being received (must be serializable with cereal)

Parameters:

source – Rank sending message
tag – MPI tag to assign to message
comm – MPI communicator message is being sent over

Returns:

Received message

template<typename T> inline T mpi_recv(int source, int tag) const

Receive an MPI message over an unspecified MPI communicator.

Template Parameters:

T – datatype being received (must be serializable with cereal)

Parameters:

source – Rank sending message
tag – MPI tag to assign to message

Returns:

Received message

template<typename T> inline T mpi_bcast(const T &to_bcast, int root, MPI_Comm comm) const

Broadcast an MPI message.

Template Parameters:

Datatype – to broadcast (must be serializable)

Parameters:

to_bcast – Data being broadcast
root – Rank message is being broadcast from
comm – MPI communicator message is being broadcast over

Returns:

Data received from root

template<typename T> inline T mpi_bcast(const T &to_bcast, int root) const

Broadcast an MPI message over an unspecified MPI communicator.

Template Parameters:

Datatype – to broadcast (must be serializable)

Parameters:

to_bcast – Data being broadcast
root – Rank message is being broadcast from

Returns:

Data received from root

inline std::ostream &cout0() const

Provides a std::cout ostream that is only writeable from rank 0.

world.cout0() << "This output is coming from rank 0" << std::endl;

Returns:: std::cout that only writes from rank 0

inline std::ostream &cerr0() const

Provides a std::cerr ostream that is only writeable from rank 0.

Returns:: std::cerr that only writes from rank 0

inline std::ostream &cout() const

Provides std::cout access with each line labeled by the rank producing the output.

Returns:: std::cout for use by any rank

inline std::ostream &cerr() const

Provides std::cerr access with each line labeled by the rank producing the output.

Returns:: std::cerr for use by any rank

template<typename ...Args> inline void cout(Args&&... args) const

python print-like function that writes to std::cout from any rank

world.cout("Printing from every rank")

Template Parameters:: Args... – Variadic argument types to print
Parameters:: args... – Variadic arguments for printing

template<typename ...Args> inline void cerr(Args&&... args) const

python print-like function that writes to std::cerr from any rank

world.cerr("Printing from every rank")

Template Parameters:: Args... – Variadic argument types to print
Parameters:: args... – Variadic arguments for printing

template<typename ...Args> inline void cout0(Args&&... args) const

python print-like function that writes to std::cout from only rank 0

world.cout0("Printing from rank 0 only")

Template Parameters:: Args... – Variadic argument types to print
Parameters:: args... – Variadic arguments for printing

template<typename ...Args> inline void cerr0(Args&&... args) const

python print-like function that writes to std::cerr from only rank 0

Template Parameters:: Args... – Variadic argument types to print
Parameters:: args... – Variadic arguments for printing

inline void enable_ygm_tracing()

Turn on tracing of YGM functions.

This is more granular than MPI tracing. YGM tracing occurs at the level of individual async calls and is indicative of the calls requested by an application. MPI tracing occurs at the level of buffers sent through YGM and is indicative of the communication YGM actually performed to meet the requests of the application’s async calls.

inline void disable_ygm_tracing(): Turn off tracing of YGM functions.

inline void enable_mpi_tracing(): Turn on tracing of MPI calls within YGM.

inline void disable_mpi_tracing(): Turn off tracing of MPI calls.

inline bool is_ygm_tracing_enabled() const

Check status of YGM tracing.

Returns:: True if currently tracing YGM functions, otherwise false

inline bool is_mpi_tracing_enabled() const

Check status of MPI tracing.

Returns:: True if currently tracing MPI calls, otherwise false

inline void set_log_level(const ygm::log_level level)

Set the log level to use in YGM.

Parameters:: level – Log level to use. Possible values in order of increasing verbosity are ygm::log_level::off, ygm::log_level::critical, ygm::log_level::error, ygm::log_level::warn, ygm::log_level::info, ygm::log_level::debug

inline log_level get_log_level()

Get the log level currently used in YGM.

Returns:: Current log level

inline void set_logger_target(const ygm::logger_target target)

Set the logger target to use in YGM.

Parameters:: target – Logger target to use. Possible values are ygm::logger_target::file, ygm::logger_target::stdout, and ygm::logger_target::stderr

inline logger_target get_logger_target()

Get the logger target currently used in YGM.

Returns:: Current logger target

template<typename ...Args> inline void log(const ygm::log_level level, Args&&... args) const

Add a message to the YGM logs.

int var = 6;
world.log(ygm::log_level::info, "This is my var: ", var);

Template Parameters:: Args... – Variadic types to add to log
Parameters:: Minimum – log level for logging message @args Variadic arguments add to log

template<typename ...Args> inline void log(const std::vector<logger_target> &targets, const ygm::log_level level, Args&&... args) const

Add a message to the YGM logs written to multiple targets.

Template Parameters:

Args... – Variadic types to add to log

Parameters:

targets – Vector of targets to write logs to
Minimum – log level for logging message @args Variadic arguments add to log

template<typename StringType> inline void set_log_location(const StringType &s)

Set the log location to use when logging to files.

Set the location of the YGM log files. One file will be created at this location for every rank.

Parameters:

s – Log location
s – Path to log location as a string

Template Parameters:

StringType – Type of provided path as string. Must be convertible to std::filesystem::path.

inline void set_log_location(std::filesystem::path p)

Set the log location to use when logging to files.

Set the location of the YGM log files. One file will be created at this location for every rank.

Parameters:

p – Log location
p – Path to log location as an std::filesystem::path

Friends

friend class detail::interrupt_mask

friend class detail::comm_stats

struct header_t

Public Members

uint32_t message_size

int32_t dest