Package org.apache.sysds.runtime.controlprogram.parfor
package org.apache.sysds.runtime.controlprogram.parfor
-
ClassDescriptionThis is the base class for all data partitioner.Partitions a given matrix into row or column partitions with a two pass-approach.MR job class for submitting parfor remote partitioning MR jobs.NOTE: for the moment we only support binary block here TODO extend impl for binarycell and textcell Interface of Writable output in order to support both PairWritableBlock and PairWritableCell.Instances of this class can be used to execute tasks in parallel.This class provides a way of dynamic task distribution to multiple workers in local multi-threaded environments.Wrapper for exchanging parfor body data structures.Super class for master/worker pattern implementations.TODO heavy hitter maintenance TODO data partitioning with binarycellWrapper for job return of ParFor REMOTE for transferring statistics and result symbol table.This class serves two purposes: (1) isolating Spark imports to enable running in environments where no Spark libraries are available, and (2) to follow the same structure as the parfor remote_mr job submission.Common functionalities for parfor workers in MR jobs.ResultMerge<T extends CacheableData<?>>Local in-memory realization of result merge.Due to independence of all iterations, any result has the following properties:A task is a logical group of one or multiple iterations (each iteration is assigned to exactly one task).This is the base class for all task partitioner.This factoring task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks.Factoring with maximum constraint (e.g., if LIX matrix out-of-core and we need to bound the maximum number of iterations per map task -> memory bounds)Factoring with minimum constraint (e.g., if communication is expensive)This naive task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to the given task size.This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.