Package org.apache.sysds.runtime.transform.tokenize.applier

Class TokenizerApplierPosition

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.applier.TokenizerApplier

org.apache.sysds.runtime.transform.tokenize.applier.TokenizerApplierPosition

All Implemented Interfaces:

Serializable

public class TokenizerApplierPosition
extends TokenizerApplier

See Also:

• Serialized Form

Field Summary

Fields inherited from class org.apache.sysds.runtime.transform.tokenize.applier.TokenizerApplier

PADDING_STRING

Constructor Summary

Constructors

Constructor	Description
TokenizerApplierPosition(int numIdCols, int maxTokens, boolean wideFormat, boolean applyPadding)

Method Summary

Modifier and Type	Method	Description
int	appendTokensLong(int row, List<Object> keys, List<Token> tokenList, FrameBlock out)
int	appendTokensWide(int row, List<Object> keys, List<Token> tokenList, FrameBlock out)
int	applyInternalRepresentation(DocumentRepresentation[] internalRepresentation, FrameBlock out, int inputRowStart, int blk)
int	getNumRows(DocumentRepresentation[] internalRepresentation)
Types.ValueType[]	getOutSchema()

Methods inherited from class org.apache.sysds.runtime.transform.tokenize.applier.TokenizerApplier

allocateInternalMeta, applyInternalRepresentation, build, getApplyTasks, getBuildTasks, getMaxNumRows, getMaxTokens, getNumCols, getOutputRow, getOutputRow, hasPadding, isWideFormat

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenizerApplierPosition

public TokenizerApplierPosition(int numIdCols,
 int maxTokens,
 boolean wideFormat,
 boolean applyPadding)

Method Details

getNumRows

public int getNumRows(DocumentRepresentation[] internalRepresentation)

Specified by:

getNumRows in class TokenizerApplier

applyInternalRepresentation

public int applyInternalRepresentation(DocumentRepresentation[] internalRepresentation,
 FrameBlock out,
 int inputRowStart,
 int blk)

appendTokensLong

public int appendTokensLong(int row,
 List<Object> keys,
 List<Token> tokenList,
 FrameBlock out)

appendTokensWide

public int appendTokensWide(int row,
 List<Object> keys,
 List<Token> tokenList,
 FrameBlock out)

getOutSchema

public Types.ValueType[] getOutSchema()

Specified by:

getOutSchema in class TokenizerApplier