Package org.apache.sysds.runtime.transform.tokenize

Class Tokenizer

java.lang.Object
org.apache.sysds.runtime.transform.tokenize.Tokenizer
All Implemented Interfaces:
Serializable
public class Tokenizer extends Object implements Serializable
See Also:

  • Field Details

    • TOKENIZE_NUM_BLOCKS

      public static final int TOKENIZE_NUM_BLOCKS

  • Method Details

    • getSchema

      public Types.ValueType[] getSchema()

    • getMaxNumRows

      public int getMaxNumRows(int inRows)

    • getNumRowsEstimate

      public int getNumRowsEstimate()

    • getNumCols

      public long getNumCols()

    • allocateInternalRepresentation

      public void allocateInternalRepresentation(int numDocuments)

    • tokenize

      public FrameBlock tokenize(FrameBlock in)

    • tokenize

      public FrameBlock tokenize(FrameBlock in, int k)

    • apply

      public FrameBlock apply(FrameBlock out, int k)

    • getBuildTasks

      public List<DependencyTask<?>> getBuildTasks(FrameBlock in)

    • build

      public void build(FrameBlock in, int k)