Package org.apache.sysds.runtime.controlprogram.paramserv.dp

Class DataPartitionerSparkMapper

java.lang.Object

org.apache.sysds.runtime.controlprogram.paramserv.dp.DataPartitionerSparkMapper

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>

public class DataPartitionerSparkMapper
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
DataPartitionerSparkMapper(Statement.PSScheme scheme, int workersNum, SparkExecutionContext sec, int numEntries)

Method Summary

Modifier and Type	Method	Description
Iterator<scala.Tuple2<Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>>	call(scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>> input)	Do data partitioning

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DataPartitionerSparkMapper

public DataPartitionerSparkMapper(Statement.PSScheme scheme,
 int workersNum,
 SparkExecutionContext sec,
 int numEntries)

Method Details

call

public Iterator<scala.Tuple2<Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>> call(scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>> input)
                                                                                              throws Exception

Do data partitioning

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>

Parameters:

input - RowBlockID => (features, labels)

Returns:

WorkerID => (rowBlockID, (single row features, single row labels))

Throws:

Exception - Some exception