Package org.apache.sysds.runtime.instructions.spark.utils

Class RDDConverterUtilsExt

java.lang.Object

org.apache.sysds.runtime.instructions.spark.utils.RDDConverterUtilsExt

public class RDDConverterUtilsExt
extends Object

NOTE: These are experimental converter utils. Once thoroughly tested, they can be moved to RDDConverterUtils.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	RDDConverterUtilsExt.AddRowID
static enum	RDDConverterUtilsExt.RDDConverterTypes

Constructor Summary

Constructors

Constructor	Description
RDDConverterUtilsExt()

Method Summary

Modifier and Type	Method	Description
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	addIDToDataFrame(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, org.apache.spark.sql.SparkSession sparkSession, String nameOfCol)	Add element indices as new column to DataFrame
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>	coordinateMatrixToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input, DataCharacteristics mcIn, boolean outputEmptyBlocks)
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>	coordinateMatrixToBinaryBlock(org.apache.spark.SparkContext sc, org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input, DataCharacteristics mcIn, boolean outputEmptyBlocks)
static void	copyRowBlocks(MatrixBlock mb, int rowIndex, MatrixBlock ret, int numRowsPerBlock, int rlen, int clen)
static void	copyRowBlocks(MatrixBlock mb, int rowIndex, MatrixBlock ret, long numRowsPerBlock, long rlen, long clen)
static void	copyRowBlocks(MatrixBlock mb, long rowIndex, MatrixBlock ret, int numRowsPerBlock, int rlen, int clen)
static void	copyRowBlocks(MatrixBlock mb, long rowIndex, MatrixBlock ret, long numRowsPerBlock, long rlen, long clen)
static void	postProcessAfterCopying(MatrixBlock ret)
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	projectColumns(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, ArrayList<String> columns)
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	stringDataFrameToVectorDataFrame(org.apache.spark.sql.SparkSession sparkSession, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> inputDF)	Convert a dataframe of comma-separated string rows to a dataframe of ml.linalg.Vector rows.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RDDConverterUtilsExt

public RDDConverterUtilsExt()

Method Details

coordinateMatrixToBinaryBlock

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> coordinateMatrixToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
 org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
 DataCharacteristics mcIn,
 boolean outputEmptyBlocks)

coordinateMatrixToBinaryBlock

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> coordinateMatrixToBinaryBlock(org.apache.spark.SparkContext sc,
 org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
 DataCharacteristics mcIn,
 boolean outputEmptyBlocks)

projectColumns

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> projectColumns(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
 ArrayList<String> columns)

copyRowBlocks

public static void copyRowBlocks(MatrixBlock mb,
 int rowIndex,
 MatrixBlock ret,
 int numRowsPerBlock,
 int rlen,
 int clen)

copyRowBlocks

public static void copyRowBlocks(MatrixBlock mb,
 long rowIndex,
 MatrixBlock ret,
 int numRowsPerBlock,
 int rlen,
 int clen)

copyRowBlocks

public static void copyRowBlocks(MatrixBlock mb,
 int rowIndex,
 MatrixBlock ret,
 long numRowsPerBlock,
 long rlen,
 long clen)

copyRowBlocks

public static void copyRowBlocks(MatrixBlock mb,
 long rowIndex,
 MatrixBlock ret,
 long numRowsPerBlock,
 long rlen,
 long clen)

postProcessAfterCopying

public static void postProcessAfterCopying(MatrixBlock ret)

addIDToDataFrame

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> addIDToDataFrame(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
 org.apache.spark.sql.SparkSession sparkSession,
 String nameOfCol)

Add element indices as new column to DataFrame

Parameters:

df - input data frame

sparkSession - the Spark Session

nameOfCol - name of index column

Returns:

new data frame

stringDataFrameToVectorDataFrame

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> stringDataFrameToVectorDataFrame(org.apache.spark.sql.SparkSession sparkSession,
 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> inputDF)

Convert a dataframe of comma-separated string rows to a dataframe of ml.linalg.Vector rows.

Example input rows:
((1.2, 4.3, 3.4))
(1.2, 3.4, 2.2)
[[1.2, 34.3, 1.2, 1.25]]
[1.2, 3.4]


Parameters:

sparkSession - Spark Session

inputDF - dataframe of comma-separated row strings to convert to dataframe of ml.linalg.Vector rows

Returns:

dataframe of ml.linalg.Vector rows