ygm::io::ndjson_parser

class ndjson_parser : public ygm::container::detail::base_iteration_value<ndjson_parser, std::tuple<boost::json::object>>

Parser for handling collections of newline-delimited JSON files in parallel.

Public Types

using for_all_args = std::tuple<boost::json::object>

using value_type = typename std::tuple_element<0, for_all_args>::type

Public Functions

template<typename ...Args> inline ndjson_parser(Args&&... args)

template<typename Function> inline void for_all(Function fn)

Executes a user function for every CSV record in a set of files.

Template Parameters:: Function –
Parameters:: fn – User function to execute

inline ygm::comm &comm()

inline const ygm::comm &comm() const

inline size_t num_invalid_records()

inline void for_all(Function &&fn)

Iterates over all items in a container and executes a user-provided function object on each.

The user-provided function is expected to take a single argument that is an item within the container. If the user provides a lambda as their function object, the lambda is allowed to capture.

Template Parameters:: Function – Type of user-provided function
Parameters:: fn – User-provided function

inline void for_all(Function &&fn) const

Const version of for_all that iterates over all items and passes them to the user function as const *.

The user-provided function is expected to take a single argument that is an item within the container. If the user provides a lambda as their function object, the lambda is allowed to capture.

Template Parameters:: Function – Type of user-provided function
Parameters:: fn – User-provided function

inline void gather(STLContainer &gto, int rank) const

Gather all values in an STL container.

Template Parameters:

STLContainer – Type of STL container to gather to

Parameters:

gto – Container to store results in
rank – Rank to tather results on. Use -1 to gather to all ranks

inline void gather(STLContainer &gto) const

Gather all values in an STL container on all ranks.

Equivalent to gather(gto, -1)

Template Parameters:: STLContainer – Type of STL container to gather to
Parameters:: gto – Container to store results in

inline std::vector<value_type> gather_topk(size_t k, Compare comp = std::greater<value_type>()) const

Gather the k “largest” values according to provided comparison function.

Template Parameters:

Compare – Type of comparison operator

Parameters:

k – Number of values to gather
comp – Comparison function for identifying elements to gather

Returns:

vector of largest values

inline value_type reduce(MergeFunction merge) const

Perform a reduction over all items in container.

reduce only makes sense to use with commutative and associative functors defining merges. Otherwise, ranks will not receive the same result.

Template Parameters:: MergeFunction – Merge functor type
Parameters:: merge – Functor to combine pairs of items
Returns:: Value from all reductions

inline void collect(YGMContainer &c) const

Collects all items in a new YGM container.

Template Parameters:: YGMContainer – Container type
Parameters:: c – Container to collect into

inline void reduce_by_key(MapType &map, ReductionOp reducer) const

Reduces all values in key-value pairs with matching keys.

Template Parameters:

MapType – Result YGM container type
ReductionOp – Functor type

Parameters:

map – YGM container to hold result
reducer – Functor for combining values

transform_proxy_value<ndjson_parser, TransformFunction> transform(TransformFunction &&ffn)

Creates proxy that transforms items in container that are presented to user for_all calls.

The underlying items within the container are not modified.

ygm::container::bag<int> my_bag(world);
my_bag.async_insert(2);
my_bag.barrier();

my_bag.transform([](auto &val) { return 2*val; }).for_all([](const auto
&transformed_val) { YGM_ASSERT_RELEASE(val == 4);
});

my_bag.for_all([](const auto &val) { YGM_ASSERT_RELEASE(val == 2); });

will complete successfully.

Template Parameters:: TransformFunction – functor type
Parameters:: ffn – Function to transform items in container

inline flatten_proxy_value<ndjson_parser> flatten()

Flattens STL containers of values to allow a function to be called on inner items individually.

Underlying container is not modified.

ygm::container::bag<std::vector<int>> my_bag(world, {{1, 2, 3}});

my_bag.flatten().for_all([](const int &nested_val) {
std::cout << "Nested value: " << nested_val << std::cout;
});

will print

Nested value: 1
Nested value: 2
Nested value: 3

filter_proxy_value<ndjson_parser, FilterFunction> filter(FilterFunction &&ffn)

Filters items in a container so only allow for_all to execute on those that satisfy a given predicate function.

Filtered items are not removed from underlying container.

ygm::container::bag<int> my_bag(world, {1, 2, 3, 4});
my_bag.filter([](const auto &val) { return (val % 2) == 0;
}).for_all([](const auto &filtered_val) { YGM_ASSERT_RELEASE((filtered_val %
2) == 0);
});

Template Parameters:: FilterFunction – Functor type
Parameters:: ffn – Function used to filter items in container.