Package org.apache.sysds.runtime.instructions.spark

Class MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction

java.lang.Object

org.apache.sysds.runtime.instructions.spark.MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

Enclosing class:

MultiReturnParameterizedBuiltinSPInstruction

public static class MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

This function pre-aggregates distinct values of recoded columns per partition (part of distributed recode map construction, used for recoding, binning and dummy coding). We operate directly over schema-specific objects to avoid unnecessary string conversion, as well as reduce memory overhead and shuffle.

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
TransformEncodeBuildFunction(MultiColumnEncoder encoder)

Method Summary

Modifier and Type	Method	Description
Iterator<scala.Tuple2<Integer,Object>>	call(Iterator<scala.Tuple2<Long,FrameBlock>> iter)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TransformEncodeBuildFunction

public TransformEncodeBuildFunction(MultiColumnEncoder encoder)

Method Details

call

public Iterator<scala.Tuple2<Integer,Object>> call(Iterator<scala.Tuple2<Long,FrameBlock>> iter)
                                            throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<scala.Tuple2<Long,FrameBlock>>,Integer,Object>

Throws:

Exception