Package org.apache.sysds.api.mlcontext

Class MLContextUtil

java.lang.Object
org.apache.sysds.api.mlcontext.MLContextUtil
public final class MLContextUtil extends Object
Utility class containing methods for working with the MLContext API.

  • Field Details

    • VERSION_NOT_AVAILABLE

      public static final String VERSION_NOT_AVAILABLE
      Version not available message.
      See Also:

    • BUILD_TIME_NOT_AVAILABLE

      public static final String BUILD_TIME_NOT_AVAILABLE
      Build time not available message.
      See Also:

    • BASIC_DATA_TYPES

      public static final Class[] BASIC_DATA_TYPES
      Basic data types supported by the MLContext API.

    • COMPLEX_DATA_TYPES

      public static final Class[] COMPLEX_DATA_TYPES
      Complex data types supported by the MLContext API.

    • ALL_SUPPORTED_DATA_TYPES

      public static final Class[] ALL_SUPPORTED_DATA_TYPES
      All data types supported by the MLContext API.

  • Constructor Details

    • MLContextUtil

      public MLContextUtil()

  • Method Details

    • isSparkVersionSupported

      public static boolean isSparkVersionSupported(String sparkVersion, String minimumRecommendedSparkVersion)
      Determine whether the Spark version is supported.
      Parameters:
      sparkVersion - Spark version string (ie, "1.5.0").
      minimumRecommendedSparkVersion - Minimum recommended Spark version string (ie, "2.1.0").
      Returns:
      true if Spark version supported; otherwise false.

    • verifySparkVersionSupported

      public static void verifySparkVersionSupported(org.apache.spark.sql.SparkSession spark)
      Check that the Spark version is supported. If it isn't supported, throw an MLContextException.
      Parameters:
      spark - SparkSession
      Throws:
      MLContextException - thrown if Spark version isn't supported

    • setDefaultConfig

      public static void setDefaultConfig()
      Set default SystemDS configuration properties.

    • setConfig

      public static void setConfig(String configFilePath)
      Set SystemDS configuration properties based on a configuration file.
      Parameters:
      configFilePath - Path to configuration file.
      Throws:
      MLContextException - if configuration file was not found or a parse exception occurred

    • setCompilerConfig

      public static void setCompilerConfig()
      Set SystemDS compiler configuration properties for MLContext

    • checkInputValueTypes

      public static void checkInputValueTypes(Map<String,Object> inputs)
      Verify that the types of input values are supported.
      Parameters:
      inputs - Map of String/Object pairs
      Throws:
      MLContextException - if an input value type is not supported

    • checkInputValueType

      public static void checkInputValueType(String name, Object value)
      Verify that the type of input value is supported.
      Parameters:
      name - The name of the input
      value - The value of the input
      Throws:
      MLContextException - if the input value type is not supported

    • checkInputParameterType

      public static void checkInputParameterType(String parameterName, Object parameterValue)
      Verify that the type of input parameter value is supported.
      Parameters:
      parameterName - The name of the input parameter
      parameterValue - The value of the input parameter
      Throws:
      MLContextException - if the input parameter value type is not supported

    • isBasicType

      public static boolean isBasicType(Object object)
      Is the object one of the supported basic data types? (Integer, Boolean, Double, String)
      Parameters:
      object - the object type to be examined
      Returns:
      true if type is a basic data type; otherwise false.

    • getBasicTypeString

      public static String getBasicTypeString(Object object)
      Obtain the SystemDS scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)
      Parameters:
      object - the object type to be examined
      Returns:
      a String representing the type as a SystemDS scalar value type

    • isComplexType

      public static boolean isComplexType(Object object)
      Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, Matrix, double[][], MatrixBlock, URL)
      Parameters:
      object - the object type to be examined
      Returns:
      true if type is a complex data type; otherwise false.

    • convertInputParametersForParser

      public static Map<String,String> convertInputParametersForParser(Map<String,Object> basicInputParameterMap)
      Converts non-string basic input parameter values to strings to pass to the parser.
      Parameters:
      basicInputParameterMap - map of input parameters
      Returns:
      map of String/String name/value pairs

    • convertInputType

      public static Data convertInputType(String parameterName, Object parameterValue)
      Convert input types to internal SystemDS representations
      Parameters:
      parameterName - The name of the input parameter
      parameterValue - The value of the input parameter
      Returns:
      input in SystemDS data representation

    • convertInputType

      public static Data convertInputType(String parameterName, Object parameterValue, Metadata metadata)
      Convert input types to internal SystemDS representations
      Parameters:
      parameterName - The name of the input parameter
      parameterValue - The value of the input parameter
      metadata - matrix/frame metadata
      Returns:
      input in SystemDS data representation

    • isCSVLineAllNumbers

      public static boolean isCSVLineAllNumbers(String line)
      If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)
      Parameters:
      line - a line of the RDD
      Returns:
      true if all the csv-separated values are numbers, false otherwise

    • doesDataFrameLookLikeMatrix

      public static boolean doesDataFrameLookLikeMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
      Examine the DataFrame schema to determine whether the data appears to be a matrix.
      Parameters:
      df - the DataFrame
      Returns:
      true if the DataFrame appears to be a matrix, false otherwise

    • quotedString

      public static String quotedString(String str)
      Return a double-quoted string with inner single and double quotes escaped.
      Parameters:
      str - the original string
      Returns:
      double-quoted string with inner single and double quotes escaped

    • displayMap

      public static String displayMap(String mapName, Map<String,Object> map)
      Display the keys and values in a Map
      Parameters:
      mapName - the name of the map
      map - Map of String keys and Object values
      Returns:
      the keys and values in the Map as a String

    • displaySet

      public static String displaySet(String setName, Set<String> set)
      Display the values in a Set
      Parameters:
      setName - the name of the Set
      set - Set of String values
      Returns:
      the values in the Set as a String

    • displaySymbolTable

      public static String displaySymbolTable(String name, LocalVariableMap symbolTable)
      Display the keys and values in the symbol table
      Parameters:
      name - the name of the symbol table
      symbolTable - the LocalVariableMap
      Returns:
      the keys and values in the symbol table as a String

    • displaySymbolTable

      public static String displaySymbolTable(LocalVariableMap symbolTable)
      Display the keys and values in the symbol table
      Parameters:
      symbolTable - the LocalVariableMap
      Returns:
      the keys and values in the symbol table as a String

    • determineOutputTypeAsString

      public static String determineOutputTypeAsString(LocalVariableMap symbolTable, String outputName)
      Obtain a symbol table output type as a String
      Parameters:
      symbolTable - the symbol table
      outputName - the name of the output variable
      Returns:
      the symbol table output type for a variable as a String

    • displayInputs

      public static String displayInputs(String name, Map<String,Object> map, LocalVariableMap symbolTable)
      Obtain a display of script inputs.
      Parameters:
      name - the title to display for the inputs
      map - the map of inputs
      symbolTable - the symbol table
      Returns:
      the script inputs represented as a String

    • displayOutputs

      public static String displayOutputs(String name, Set<String> outputNames, LocalVariableMap symbolTable)
      Obtain a display of the script outputs.
      Parameters:
      name - the title to display for the outputs
      outputNames - the names of the output variables
      symbolTable - the symbol table
      Returns:
      the script outputs represented as a String

    • displayOutputs

      public static String displayOutputs(Set<String> outputNames, LocalVariableMap symbolTable)
      Obtain a display of the script outputs.
      Parameters:
      outputNames - the names of the output variables
      symbolTable - the symbol table
      Returns:
      the script outputs represented as a String

    • welcomeMessage

      public static String welcomeMessage()
      The SystemDS welcome message
      Returns:
      the SystemDS welcome message

    • getSparkContext

      public static org.apache.spark.SparkContext getSparkContext(MLContext mlContext)
      Obtain the Spark Context
      Parameters:
      mlContext - the SystemDS MLContext
      Returns:
      the Spark Context

    • getJavaSparkContext

      public static org.apache.spark.api.java.JavaSparkContext getJavaSparkContext(MLContext mlContext)
      Obtain the Java Spark Context
      Parameters:
      mlContext - the SystemDS MLContext
      Returns:
      the Java Spark Context

    • getSparkContextFromProxy

      public static org.apache.spark.SparkContext getSparkContextFromProxy()
      Obtain the Spark Context from the MLContextProxy
      Returns:
      the Spark Context

    • getJavaSparkContextFromProxy

      public static org.apache.spark.api.java.JavaSparkContext getJavaSparkContextFromProxy()
      Obtain the Java Spark Context from the MLContextProxy
      Returns:
      the Java Spark Context

    • getSparkSessionFromProxy

      public static org.apache.spark.sql.SparkSession getSparkSessionFromProxy()
      Obtain the Spark Session from the MLContextProxy
      Returns:
      the Spark Session

    • doesSymbolTableContainFrameObject

      public static boolean doesSymbolTableContainFrameObject(LocalVariableMap symbolTable, String variableName)
      Determine if the symbol table contains a FrameObject with the given variable name.
      Parameters:
      symbolTable - the LocalVariableMap
      variableName - the variable name
      Returns:
      true if the variable in the symbol table is a FrameObject, false otherwise.

    • doesSymbolTableContainMatrixObject

      public static boolean doesSymbolTableContainMatrixObject(LocalVariableMap symbolTable, String variableName)
      Determine if the symbol table contains a MatrixObject with the given variable name.
      Parameters:
      symbolTable - the LocalVariableMap
      variableName - the variable name
      Returns:
      true if the variable in the symbol table is a MatrixObject, false otherwise.

    • deleteRemoveVariableInstructions

      public static void deleteRemoveVariableInstructions(Program progam)
      Delete the 'remove variable' instructions from a runtime program.
      Parameters:
      progam - runtime program